How do I use the robots.txt tester?
- Enter the full page URL to test.
- Choose the crawler: Googlebot, Bingbot or an AI crawler.
- Read the result and the rule that decided it.
robots.txt tester: frequently asked questions
Does blocking in robots.txt remove a page from Google?
No. It stops crawling, but a blocked page can still be indexed from links. Use a noindex tag (and allow crawling) to remove a page.
Which rule wins if several match?
The most specific (longest) matching path wins. If Allow and Disallow are equally long, Allow wins.
How do I block AI crawlers?
Add a group for each, e.g. “User-agent: GPTBot” with “Disallow: /”. Our robots.txt generator can do it for you.
What if there is no robots.txt?
If it returns 404, crawlers treat everything as allowed.
What if robots.txt returns a server error?
Google may treat the whole site as blocked until robots.txt is reachable again, so fix 5xx errors quickly.
Do wildcards work?
Yes. * matches any characters and $ marks the end of the URL.
Is robots.txt case-sensitive?
Paths are case-sensitive; user-agent names are not.
Where must robots.txt be?
At the root of each host, e.g. https://example.ae/robots.txt. Subdomains need their own.
Will this tool improve my rankings?
It helps you find and fix issues. Rankings depend on content, technical health and authority together; our SEO team can help with the rest.
Is the robots.txt tester free?
Yes. It is free with no sign-up and no usage limits for normal use.