Use Free Tools

Robots.txt Generator

Build a valid robots.txt file without touching code. Our free generator lets you add allow and disallow rules for any crawler, set a crawl delay, and include your sitemap URL — then download a ready-to-upload file in seconds.

  • Free, no signup
  • Runs in your browser
  • Works on mobile
Start from
Rules
Your robots.txt










How to use the Robots.txt Generator
  1. Pick a starting point: Recommended, Allow everything, or Block everything.
  2. Keep the user-agent as * to address every crawler, or type a specific bot such as Googlebot.
  3. Edit, add or remove rules. Disallow blocks a path; Allow makes an exception inside a blocked folder.
  4. Add your sitemap URL and, if you want, block AI training crawlers. The file updates as you type.
  5. Download robots.txt and upload it to the root of your domain so it lives at yourdomain.com/robots.txt.


About this tool
robots.txt is a plain text file at the root of your site that tells search engine crawlers which paths they may visit. The generator writes valid syntax as you type, adds a leading slash to paths if you forget it, and can add a separate block for AI training crawlers. Remember that robots.txt controls crawling, not indexing: to keep a page out of Google, use a noindex meta tag instead.
Why use itEvery professional website needs a robots.txt: it keeps search engines out of admin panels, staging areas, and thin pages that waste your crawl budget, while pointing them to your sitemap for faster discovery of new content. A single syntax error — like a misplaced 'Disallow: /' — can accidentally block your entire site from Google, so generating correctly formatted rules matters more than most people realize.


Frequently asked questions
Where do I upload robots.txt?
It must live at the root of your domain — https://toolshacks.com/robots.txt. A robots.txt placed in any subfolder is completely ignored by crawlers.
Does robots.txt keep pages out of Google?
It blocks crawling, not indexing. A disallowed page can still appear in search results if other sites link to it — use a noindex meta tag or password protection for truly private pages.
What is crawl-delay?
It politely asks crawlers to wait a number of seconds between requests. Googlebot ignores it, but some smaller crawlers and aggressive bots respect it.
Can a robots.txt mistake hurt my site?
Absolutely — 'Disallow: /' blocks your entire site from all crawlers and can wipe you from Google within days. Always test changes in Google Search Console before deploying.