Skip to content
DevLab

🤖 robots.txt Detailed Analyzer

Just enter a hostname to fetch /robots.txt and analyze Allow / Disallow rules per User-agent group, Sitemap, Crawl-delay, and the permission status for major search engine and AI crawlers.

100% Free No signup Server-side No logs / DB Rate-limited 5 languages Dark mode

⚠️ Requests are made from DevLab servers. Private IP addresses and localhost are not allowed.

📖 What this check tells you

This check fetches /robots.txt, parses the syntax, and lists the rules per user-agent group, the permission status of the major search crawlers and AI bots, and every Sitemap: line. The first thing to internalise is that robots.txt controls crawling, not indexing. A disallowed page can still appear in results as a bare URL if another site links to it.

Item What it means How to fix it
Using Disallow to prevent indexing Disallow only stops crawling. Worse, noindex can only be seen by reading the page, so combining Disallow with noindex means the noindex is never read. The Search Console warning about a page being indexed though blocked by robots.txt describes exactly this situation. To keep something out of the index, allow it in robots.txt and return <meta name="robots" content="noindex">, or an X-Robots-Tag: noindex header for non-HTML files. Anything that must not be seen at all belongs behind authentication (401 / 403), not robots.txt — the file is public, so listing a secret path is an invitation.
A user-agent group is not taking effect A crawler obeys exactly one group — the most specific match for its own name — and does not combine it with the User-agent: * group. The moment you add a Googlebot group, Googlebot ignores every rule under *. Shared Disallow lines silently dropping out this way is a very common accident. When you create a specific group, copy the shared rules from * into it as well. Conversely, if no bot genuinely needs different treatment, keeping everything in a single * group is the safe choice. Where Allow and Disallow conflict, the longer — more specific — path wins.
The path patterns do not mean what you think Paths are matched as a case-sensitive prefix. Disallow: /admin also blocks /administrator/ and /admin-guide.html. The only wildcards are * for any sequence and $ for end-of-URL; regular expressions are not interpreted. To target only a directory, add the trailing slash (Disallow: /admin/). To filter by extension, use Disallow: /*.pdf$. Never disallow CSS or JavaScript — Google renders the page to evaluate it, so blocking them makes the layout look broken and drags down the mobile assessment too.

robots.txt is per host and per port. https://example.com/robots.txt and https://www.example.com/robots.txt are unrelated files, and every subdomain needs its own. The failure behaviour is defined too: a 404 means everything is allowed, while sustained 5xx makes Google treat the whole site as disallowed for a while. If your maintenance mode returns 503 site-wide, carve out robots.txt so it still answers 200.

🔗 Related Tools