Drag-n-Drop
to upload your archive
Robots.txt Generator
Build a valid robots.txt in seconds
Pick a preset, add your own Allow and Disallow rules, drop in your sitemap URL, and the file is written for you line by line as you type. Everything runs in your browser, so nothing about your site is ever sent to us. Copy it or download a ready-to-upload robots.txt.
Crawl rules
| User-agent | Directive | Path | Actions |
|---|
Leave the path empty for a bare Disallow: — that is the canonical way to say “crawl everything”. Use / to block the whole site.
Optional directives
Honoured by Bing, Yandex and Seznam. Google ignores it — set crawl rate in Search Console instead.
A Yandex-specific hint for your preferred domain. Most sites can leave this blank.
Must be absolute URLs. They are written at the very bottom of the file, outside every user-agent group — that is the convention crawlers expect.
-
Correct grouping, automatically
Rules for the same user-agent are merged into one block, blocks are separated by a blank line, and Sitemap lines go last. The three things hand-written robots.txt files get wrong most often.
-
One-click AI crawler block
Twenty named AI and LLM agents — GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider and more — blocked in a single click, without touching Googlebot or Bingbot.
-
Live preview, live warnings
The file rewrites itself on every keystroke, and the status line flags the classics: a path missing its leading slash, a relative sitemap URL, an empty user-agent.
How to create a robots.txt file
-
1
Pick a preset, then add or remove rules until the crawl rules match your site.
-
2
Paste your sitemap URL and, if you need it, a crawl delay.
-
3
Download the file and upload it to your site root, so it answers at /robots.txt.
What a robots.txt file actually controls
-
Crawling, not indexing
Disallow tells a crawler not to fetch a URL. It does not remove that URL from search results — a blocked page can still be listed if other sites link to it. To keep a page out of the index, let it be crawled and serve a
noindexmeta tag or header. -
Crawl budget on big sites
Faceted navigation, internal search results and endless filter combinations can burn a crawler's time on URLs that will never rank. Blocking those patterns points the budget at the pages you actually care about. On a small brochure site this matters far less than people think.
-
A public, voluntary standard
Anyone can read your robots.txt, and compliance is opt-in. Well-behaved crawlers follow it; scrapers and spam bots ignore it entirely. Never list secret paths there, and never treat it as access control — use authentication for anything that must stay private.
Got your robots.txt? Publish the whole site with free static hosting
Select File
or drop your archive to upload
Frequently asked questions
What is a robots.txt file?
robots.txt is a plain text file at the root of your domain that tells crawlers which parts of your site they may request. It is the first thing most well-behaved bots fetch. It is a set of instructions, not a security boundary: it is publicly readable and hostile scrapers simply ignore it, so never use it to hide sensitive URLs.
Where does robots.txt go?
It must live at the root of the host, at https://example.com/robots.txt, and it only applies to that exact host and protocol. A file at /blog/robots.txt does nothing, and a file on www.example.com does not cover example.com.
What is the difference between Disallow and noindex?
Disallow stops a crawler from fetching a URL; it does not remove that URL from search results. A blocked page can still be indexed from external links, appearing with no snippet. To keep a page out of the index you need a noindex robots meta tag, and the crawler has to be allowed to fetch the page to see it. Blocking in robots.txt and adding noindex at the same time cancels itself out.
How do I block AI crawlers?
Add a Disallow rule for each AI user agent you want to exclude. The "Block AI crawlers" preset here covers the current major ones, including GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider and meta-externalagent. Note that Google-Extended controls AI training use without affecting normal Google Search crawling, which is usually what site owners actually want.
What is the Content-Signal line?
Content Signals is a newer, machine readable way to express intent per use case rather than per bot, for example allowing search indexing while declining AI training. We ship one in our own robots.txt. It does not replace Disallow rules and adoption is still uneven, so treat it as a clear statement of intent that sits alongside your explicit rules.
Should I list my sitemap in robots.txt?
Yes. A Sitemap: line with the full absolute URL is read by every major engine and costs nothing. It is the simplest way to make sure crawlers find your sitemap even if you never submit it manually. Build one with our XML Sitemap Generator.
Does crawl-delay work with Google?
No. Google ignores Crawl-delay entirely and adjusts its crawl rate automatically. Bing and Yandex do honour it. Only add it if a specific bot is genuinely overloading your server, and prefer fixing the underlying performance problem instead.
How do I test my robots.txt?
Upload the file, open it in a browser to confirm it is served as plain text at the root, then use the robots.txt report in Google Search Console to see how Google parses it. If you host with Static.app, dropping robots.txt into your project root is all that is needed: it is served as a static file, no configuration required.