Build a valid robots.txt file without touching code. Our free generator lets you add allow and disallow rules for any crawler, set a crawl delay, and include your sitemap URL — then download a ready-to-upload file in seconds.
Your robots.txt will appear here...
A robots.txt file tells search engine crawlers which parts of your site they may or may not crawl. Place the generated file at your domain root (e.g. https://example.com/robots.txt).
Every professional website needs a robots.txt: it keeps search engines out of admin panels, staging areas, and thin pages that waste your crawl budget, while pointing them to your sitemap for faster discovery of new content. A single syntax error — like a misplaced 'Disallow: /' — can accidentally block your entire site from Google, so generating correctly formatted rules matters more than most people realize.
It must live at the root of your domain — https://toolshacks.com/robots.txt. A robots.txt placed in any subfolder is completely ignored by crawlers.
It blocks crawling, not indexing. A disallowed page can still appear in search results if other sites link to it — use a noindex meta tag or password protection for truly private pages.
It politely asks crawlers to wait a number of seconds between requests. Googlebot ignores it, but some smaller crawlers and aggressive bots respect it.
Absolutely — 'Disallow: /' blocks your entire site from all crawlers and can wipe you from Google within days. Always test changes in Google Search Console before deploying.