How do I use the robots.txt generator?
- Enter your sitemap URL.
- List paths to keep out of search, one per line.
- Choose whether to allow AI crawlers, then build and upload the file to your site root.
robots.txt generator: frequently asked questions
Does robots.txt hide a page from Google?
No. It stops crawling, but a blocked URL can still appear in results if other sites link to it. Use a noindex tag to keep a page out.
Should I allow AI crawlers?
If you want ChatGPT, Claude and Perplexity to quote and link to you, yes. Blocking them keeps your content out of AI answers.
What is robots.txt?
A file at your site root telling crawlers which parts of the site they may crawl.
Does robots.txt hide pages from Google?
No. Blocked pages can still be indexed if linked. Use a noindex tag to keep a page out of results.
What should a typical robots.txt block?
Admin areas, internal search results, cart and checkout steps, and staging paths — never CSS or JavaScript that pages need.
Should I add my sitemap to robots.txt?
Yes. Add a Sitemap: line with the full URL of your sitemap or sitemap index.
Can I block AI crawlers?
Yes, by user agent, for example GPTBot or Google-Extended. Blocking them may reduce how often AI assistants cite you.
What does Disallow: / do?
It blocks the whole site. It is common on staging sites and a disaster on live ones.
Where must robots.txt be located?
At the root of each host, such as https://example.com/robots.txt.
How do I test robots.txt?
Use the robots.txt report in Google Search Console after publishing.