Named groups written properly
A crawler obeys one group only, so a named group replaces your wildcard rules rather than adding to them. This repeats the rules that still apply into every group that needs them.
Free tool · No signup
Generate a robots.txt file
Pick what each kind of crawler may read and get a correct file, with the named groups written out properly so your rules do not silently stop applying.
robots.txt generator Free
- Separates AI training from AI search
- Repeats rules into named groups
- No signup
YOUR FILE
Copy this to your site root
Save it as robots.txt so it resolves at yoursite.com/robots.txt
WHAT THIS FILE DOES
BEFORE YOU SHIP IT
Test it against a real URL.
A robots.txt file that looks right and behaves differently is the normal outcome, because precedence is not intuitive. Check a path against the file before you rely on it.
MANAGING MANY SITES?
Audit robots.txt across a portfolio.
Writing one file is a five-minute job. Checking that fifty sites still serve the rules you intended is not. The Web Scraping API returns any site's robots.txt as structured data, so the audit runs on a schedule instead of by hand.
WHAT THIS GENERATOR DOES DIFFERENTLY
Training and search kept apart
Blocking GPTBot costs no traffic. Blocking OAI-SearchBot removes you from AI answers. Most generators lump them together as "AI bots".
Tells you what it did
Every file comes with plain-language notes on what the rules actually permit, including the costs of the stricter options.
ROBOTS.TXT GENERATOR GUIDE
A short file that is easy to get wrong.
robots.txt is four directives and a handful of rules, and almost every mistake comes from the same two misunderstandings. This guide covers how to install the file on common platforms and what the rules actually mean.
Where the robots.txt file goes
At the root of the host, so it resolves at https://example.com/robots.txt. Not in a subfolder, and not renamed. A crawler requests exactly that path and nothing else.
Every host needs its own file. Rules on example.com do nothing for docs.example.com or shop.example.com, which is one of the most common reasons a section stays crawlable when someone thought they had blocked it.
After uploading, load the URL in a browser. If you see your 404 page, the file is not where a crawler will look.
How to add robots.txt to WordPress
WordPress serves a virtual robots.txt: if no real file exists, it generates a minimal one on the fly. That is why editing a file you cannot find sometimes appears to do nothing.
Two routes work:
- Upload a real file. Put
robots.txtin the web root, usuallypublic_html, using FTP or your host's file manager. A real file always wins over the virtual one. - Use your SEO plugin. Yoast (Tools, then File editor), Rank Math (General Settings, then Edit robots.txt) and All in One SEO (Tools, then Robots.txt Editor) all write the rules for you.
Also check Settings, then Reading: the "Discourage search engines from indexing this site" box overrides everything and is a frequent cause of a site quietly disappearing.
How to add robots.txt to Blogger
Blogger does not let you upload files, so paste the rules instead. Go to Settings, scroll to Crawlers and indexing, turn on Enable custom robots.txt, then paste the generated file and save.
Keep the Sitemap: line pointing at https://yourblog.blogspot.com/sitemap.xml, or your custom domain if you use one.
Shopify, Wix and other hosted platforms
Shopify generates robots.txt for you and allows edits through the robots.txt.liquid theme template. Wix and Squarespace expose an editor in their SEO settings. In each case the platform owns the file, so paste rules rather than uploading.
Be careful about removing a platform's default rules. They usually block cart, checkout and internal search URLs for good reason.
Choosing an AI crawler policy
This is the choice worth thinking about, because the two kinds of AI crawler have opposite consequences.
Training crawlers such as GPTBot, ClaudeBot and CCBot gather text that may contribute to a model. They send no traffic back, so blocking them costs you nothing in visitors.
AI search crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot fetch pages so an assistant can answer a question, usually with a link. Blocking these removes you from AI answers entirely.
The default here blocks the first group and allows the second, which is the right trade for most sites that want to be found. Read the full guide to blocking AI crawlers if you want the reasoning and the complete user-agent list.
Common mistakes this generator avoids
A named group silently discarding your rules. A crawler obeys only the group that best matches its name, so adding User-agent: GPTBot means GPTBot stops reading everything under *. Rules that still apply have to be repeated. This generator does that for you.
Blocking CSS and JavaScript. Disallowing /assets/ or /wp-content/ stops a renderer fetching what the page needs, so it is judged on a broken version of itself.
A trailing slash that changes the meaning. Disallow: /admin also blocks /administrator. Disallow: /admin/ does not.
Using robots.txt to hide something. The file is public and lists the paths you would rather nobody visited. Anything genuinely private needs authentication.
Blocking a page to remove it from Google. A blocked page cannot be read, so its noindex is never seen and the URL can stay listed. Allow the crawl and serve noindex instead.
robots.txt questions
Where do I put the robots.txt file?
At the root of the host, so it resolves at https://example.com/robots.txt. Every host needs its own file.
How do I create a robots.txt file for WordPress?
Upload a real file to the web root, or paste the rules into the robots.txt editor in Yoast, Rank Math or All in One SEO. A real file overrides the virtual one WordPress serves.
Can I add a robots.txt file to Blogger?
Yes. Settings, then Crawlers and indexing, then enable custom robots.txt and paste the rules.
Should I block AI crawlers?
Block training crawlers if you would rather not contribute to model training; it costs no traffic. Think harder before blocking AI search crawlers, because that removes you from AI answers.
Does robots.txt stop a page being indexed?
No. It asks crawlers not to fetch it. Use noindex to keep a page out of an index.
Why did one new rule break my other rules?
A crawler obeys one group only. A named group replaces the wildcard group for that crawler, so rules it still needs must be repeated inside it.
Generate it, then prove it works
A generated file is a starting point. The only way to know a rule does what you meant is to test a real URL against it.
These free tools check the rest of the same surface.
Test a URL
Check whether a path is allowed for any crawler with the robots.txt Tester.
Check AI crawlers
See every AI bot's verdict at once with the AI Crawler Access Checker.
Declare a sitemap
Validate the sitemap you point at with the XML Sitemap Checker.