Check which AI crawlers can read your site

The AI crawlers this checker tests

Not every AI crawler does the same job, and the distinction changes what blocking one costs you.

Training crawlers

GPTBot from OpenAI, ClaudeBot from Anthropic, Google-Extended, Applebot-Extended, meta-externalagent, CCBot and Bytespider gather content used to train models. Blocking these keeps your writing out of a training set. It does not remove you from anything a user sees today.

Search and answer crawlers

OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes those assistants search when answering. Blocking these removes you from their answers, which for most sites is the opposite of the goal.

User-triggered fetchers

ChatGPT-User, Claude-User and Perplexity-User fetch a page at the moment a person asks about it. Blocking these means that when somebody pastes your URL into an assistant and asks what it says, the assistant cannot read it.

How this AI crawler access checker works

It fetches /robots.txt from the domain you enter and parses it the way RFC 9309 specifies, then evaluates each crawler against the result.

That evaluation is less obvious than it looks, and it is where simpler checkers disagree with reality. A crawler uses the group whose user-agent value is the longest match for its own name; * is a fallback used only when no named group matches, not a rule that competes with one. Within the chosen group, the longest matching path pattern wins, and when an Allow and a Disallow match at equal length, Allow wins. An empty Disallow: means nothing is disallowed.

Because those rules decide the answer, this checker reports the specific rule and line number behind every verdict rather than a bare allowed or blocked.

How to block or allow AI bots in robots.txt

To block a single crawler, give it its own group:

User-agent: GPTBot
Disallow: /

To allow everything except one area, disallow the path rather than the whole site:

User-agent: GPTBot
Disallow: /members/
Allow: /

Each crawler needs its own group. There is no wildcard for "all AI bots", so blocking GPTBot does nothing to ClaudeBot. Place the file at your domain root, reachable at https://example.com/robots.txt, and remember rules are per host: a subdomain has its own robots.txt and is not covered by the parent domain's.

Should you block AI crawlers?

There is no universal answer, but the question is more precise than it usually gets asked. "Should I block AI?" bundles three separate decisions.

Training is a genuine judgement call. If your content is your product, keeping it out of training data is defensible. If your content exists to attract customers, being part of what models know about your field is usually an advantage.

Search and answers is rarely a good block. These crawlers decide whether you appear when someone asks an assistant a question in your category. Blocking them is closer to removing yourself from Google than to protecting your work.

User-triggered fetches are almost never worth blocking. Somebody has explicitly asked about your page; refusing to serve it means the assistant answers from memory or not at all.

The common error is blocking all three when only the first was intended.

Common robots.txt mistakes this checker catches

Blocking a crawler that does not exist. User-agent names are specific strings. A group for Claude or OpenAI matches nothing, because the real names are ClaudeBot and GPTBot. The check will show you the group that actually applied.

Assuming a wildcard covers named crawlers. A generous User-agent: * block does not apply to a crawler that has its own group, even a nearly empty one.

Forgetting subdomains. Blocking crawlers on example.com leaves docs.example.com untouched. Check each host separately.

Believing robots.txt is enforcement. It is a published request that well-behaved crawlers honour. It is not authentication, and it does not protect anything that genuinely needs protecting.

AI crawler questions