🤖 robots.txt Generator
Build a robots.txt for 20+ major crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Bytespider and more) with Allow / Disallow, Crawl-delay, Sitemap and Host directives.
🔒 About Privacy
- ・All processing happens entirely in your browser
- ・Inputs are never sent to any server
- ・No signup, no login, no payment ever
⚡ Presets
🤖 Crawlers (User-agent)
📄 robots.txt
📖 Where people get stuck
Generates a robots.txt with per-crawler Allow and Disallow rules, Crawl-delay, Sitemap and Host, including presets for AI crawlers. Everything runs in the browser. What robots.txt controls is whether a page may be crawled, not whether it appears in search results — confusing those two is behind very nearly every misunderstanding about this file.
| Case | What happens | What to do |
|---|---|---|
| A disallowed page still appears in search results | robots.txt stops crawling, not indexing. If another site links to the page, Google registers the URL without ever reading the content — and the result in the listing is a bare URL where the title should be, with a note that no information is available because of robots.txt. So rather than hiding anything, Disallow leaves a URL in the results with nothing behind it. And because the content is never read, any noindex you wrote is never seen either. |
To keep something out of the results, use noindex — either <meta name="robots" content="noindex"> in the HTML or an X-Robots-Tag: noindex header. And do not also Disallow that page in robots.txt: reading noindex requires crawling, so using both together produces the opposite of what you intended. The same applies to removing a URL that is already indexed — the correct order is lift the Disallow, let noindex be read, wait for the removal, and only then Disallow again. Anything that genuinely must not be seen belongs behind authentication: a scheme that publishes something and hides it only from search engines cannot be relied on in principle. |
| Disallow entries become a map for attackers | A robots.txt is a public file anyone can read. Writing Disallow: /admin/ or Disallow: /backup/ announces to the world that an admin panel and backups live there. And since robots.txt is only a convention, a malicious bot reads it first and goes straight for those paths — the places you meant to hide become the most efficient starting point for a search. Vulnerability scanners build fetching robots.txt into their first step for exactly this reason. |
Do not name the things you want protected in robots.txt. Admin panels, backups and internal APIs belong behind basic authentication, an IP restriction or a VPN, and that is the only real measure — put them there and there is nothing to write in robots.txt at all, since a 401 stops the crawl anyway. If you must obscure a path, naming only the parent directory rather than each specific path is an option, but it is reassurance rather than protection. The test is simple: if publishing that path would be a problem, it is not a job for robots.txt. |
| Blocking CSS and JS degrades how the page is assessed | Google evaluates a page by actually rendering it, which means loading the CSS and JavaScript exactly as a browser would. If an old habit such as Disallow: /assets/ or Disallow: /wp-includes/ is still in place, rendering fails and Google sees a page with broken layout, or with no content at all. Where the body is drawn on the client, the body itself is simply not seen. The mobile-friendly assessment also fails when the CSS cannot be read. The nature of this problem is that you notice it only once rankings have dropped. |
Always allow the resources needed to render. Checking is easy: run Test live URL in the Search Console URL inspection tool and look at the rendered screenshot and the list of blocked resources — a blank screen or an unstyled page is your answer. If a broad Disallow is already in place, you can open specific holes with Allow — Allow: /assets/*.css and Allow: /assets/*.js after Disallow: /assets/. In truth, a modern site has almost no reason to block static resources at all, so when in doubt, allow them. |
A robots.txt is only honoured at the root of a domain. example.com/blog/robots.txt is ignored entirely. Subdomains, protocols and ports all count as separate, so blog.example.com needs its own file, and in principle http:// and https:// are distinct too. On AI crawlers: GPTBot, ClaudeBot, PerplexityBot and Google-Extended are not the search crawlers, so blocking them does not affect your rankings — blocking Google-Extended leaves Googlebot crawling as before. Whether to allow them is a business decision with no technical right answer: allow them if you want traffic arriving through AI answers. Note also that the list of bot names keeps growing, so whatever you write today will be out of date in six months — treat the file as something to revisit, not something to write once. If what you want is to point AI at content worth reading, that is llms.txt, not robots.txt; they do different jobs.
📖 How to Use
-
1
Pick crawlersCheck the crawlers and toggle Allow / Disallow.
-
2
Add paths and optionsEnter Allow / Disallow paths, Crawl-delay, Sitemap URL and Host.
-
3
Copy or downloadCopy the generated robots.txt or save as file.
❓ Frequently Asked Questions
Does blocking GPTBot or ClaudeBot affect SEO ranking?
Why is my Disallow ignored?
Where should robots.txt be placed?
🔗 Related Tools
🐛 Found a bug or issue with this tool?
Free to use, no signup. Even just the steps to reproduce are helpful. Reports go directly to the operator and help us fix issues.
Thanks for your report!
Your report has been delivered to the operator and will be used to improve the tool.