🤖 robots.txt Tester
Simply input the robots.txt content and target URL path to determine whether that path is allowed for each User-agent such as Googlebot / GPTBot / ClaudeBot. Correctly parses Allow / Disallow / longest match rules.
🤖 All major crawlers at once
🔗 Related Tools
📖 Where people get stuck
Matches the robots.txt body you paste against a URL path and decides the outcome with the longest-match rule over Allow and Disallow. It also shows a bulk verdict for major crawlers such as Googlebot, GPTBot and ClaudeBot. Everything runs in the browser and it does not connect to your site — to fetch and analyze the robots.txt that is actually published, use the robots.txt analyzer instead. And, as the premise that matters most: this file controls crawling, not indexing. "I disallowed it but it will not leave the search results" is the specified behavior.
| Case | What happens | What to do |
|---|---|---|
Disallow does not mean not indexed |
What Disallow says is do not come and fetch this, not do not list this in search results. If external links point to the URL, a search engine may list it without ever looking at the content — that is the state where the description is replaced by a note saying no information is available. More serious is the ordering problem: you can put noindex on a page, but if that page is disallowed the crawler can never read the noindex. An instruction that cannot be read is not carried out. Which makes Disallow the one reliable way to guarantee your noindex never takes effect. |
Choose according to your goal. To remove something from search results, allow crawling and return a noindex — either <meta name="robots" content="noindex"> in the HTML or the X-Robots-Tag: noindex HTTP header (use the latter for PDFs and images). It takes effect when the page is next crawled, so pair it with the removal tool in Search Console if you are in a hurry. Disallow, on the other hand, is right when you want to reduce crawling itself — endless combinations of filter parameters, internal search results, calendars that run into the far future. And neither one protects anything confidential. Put authentication in front of it; everything else is merely a request. |
| A crawler reads exactly one group | This is the most misunderstood part of the specification. A crawler selects exactly one group, the one whose name matches it most specifically, and ignores every other group completely — including User-agent: *. Rules are not merged. So if you write Disallow: /admin/ under User-agent: * and later add a User-agent: Googlebot group containing only Allow: /, Googlebot ignores the * group entirely and crawls /admin/. You meant to add one exception for a specific bot, and instead removed every restriction that applied to it — that accident comes straight out of this rule. |
Once you create a group for a specific bot, copy every shared rule into that group as well. The duplication is not redundant; under this specification it is mandatory. Afterwards, always confirm with the bulk verdict on this page — it evaluates the same path down a column of major crawlers, so any single bot whose row differs is the one you forgot to copy to. There is a knack to choosing test paths too: if you only try the URLs you want blocked, a file that accidentally says Disallow: / for everything still looks like it is working correctly. Always include at least one URL you expect to be allowed — the home page, or a major article page. |
Paths match by prefix, and only * and $ are special |
The value of Disallow is compared as a string from the start of the URL path, not as a directory name. So Disallow: /admin blocks not only /admin/ but also /administrator-guide and /admin-policy.html. It is case-sensitive — /Admin/ is a different string. The only wildcards are * (any sequence) and $ (end of path); regular expressions are not available. When you target an extension the $ is mandatory, since Disallow: /*.pdf also matches /a.pdf?x=1 and /a.pdf.html. And the most expensive mistake is a single character: Disallow: with nothing after it means allow everything, while Disallow: / means block everything. |
If you mean a directory, always write the trailing slash (Disallow: /admin/). Then try both shapes on this page — evaluate /admin/ and /admin-tools/ and you will see on the spot whether the rule reaches wider than you intended. The query string is part of what is matched, so if you want to stop faceted URLs, test the form you actually use, such as Disallow: /*?. And before you publish, paste the entire production robots.txt here and confirm that the home page comes out as Allow — that single step prevents every instance of shipping a staging Disallow: / to production. |
robots.txt applies per origin. Differ in scheme, hostname or port and it is a different file — https://example.com/robots.txt does not apply to http://, to www.example.com, or to blog.example.com. It lives at the root and nowhere else; a copy in a subdirectory is never read. On syntax: Google ignores Crawl-delay (to slow it down, set the rate in Search Console). Sitemap: is the one exception that belongs to no group and is valid anywhere in the file. And finally, this is not an enforcement mechanism. Obeying it is optional for the crawler, and a malicious bot will read it and ignore it. Worse, the file publishes a list of the paths you would rather hide — writing Disallow: /secret-admin/ is the same as announcing that an admin panel is there. Do not put anything you want kept secret into robots.txt.
📖 How to Use
-
1
Paste your robots.txt contentPaste the content of your robots.txt into the left textarea. Use the Sample button to try a typical robots.txt example.
-
2
Enter the URL path and User-agent to testEnter a path such as /admin/ in the Test URL path field and select a User-agent (Googlebot, GPTBot, ClaudeBot, etc.) from the dropdown.
-
3
Review the verdict and bulk crawler checkThe verdict (ALLOWED / DISALLOWED) and matched rule are shown for the selected User-agent. The bulk table at the bottom shows results for all major crawlers at once.
❓ Frequently Asked Questions
What if both Disallow: / and Allow: / are present?
Is the Sitemap directive parsed?
Is User-agent matching case-sensitive?
🐛 Found a bug or issue with this tool?
Free to use, no signup. Even just the steps to reproduce are helpful. Reports go directly to the operator and help us fix issues.
Thanks for your report!
Your report has been delivered to the operator and will be used to improve the tool.