The AI crawlers hitting your site, and what each one is actually for

Someone at your company, probably in late 2023, read a headline about AI companies scraping the web and added this to robots.txt:

User-agent: GPTBot
Disallow: /

Reasonable at the time. The problem is that OpenAI now runs three crawlers, and that rule only stops one of them — while a lot of sites went further and blocked all three, which quietly removes them from ChatGPT's answers without removing them from anyone's training set.

The distinction nobody explains clearly: a training crawler collects pages to train future models. A search crawler builds the index an assistant queries when it answers a live question. A user fetch happens when a person pastes your URL into an assistant and it goes and reads that one page.

Blocking training costs you nothing today. Blocking search costs you every citation.

The list

These are the agents worth knowing, with the job each one does. Every entry here is checked against the operator's own documentation rather than a blog post, because the names change and the blog posts do not.

OpenAI — GPTBot (training), OAI-SearchBot (search, feeds ChatGPT's answers), ChatGPT-User (fetches a page when a user asks for it).

Anthropic — ClaudeBot (training), Claude-SearchBot (search), Claude-User (user fetch).

Perplexity — PerplexityBot (search), Perplexity-User (user fetch).

Google — Google-Extended. This one is not a crawler at all. It is a control token: it does not appear in your logs, and it only tells Google whether your content may be used for Gemini training. Googlebot still crawls you for Search, and disallowing Google-Extended does not remove you from AI Overviews.

Apple — Applebot (search, also feeds Siri), and Applebot-Extended, another control token for training opt-out.

Meta — Meta-ExternalAgent (training), Meta-ExternalFetcher (user fetch).

Others — Amazonbot, CCBot (Common Crawl, whose archives end up in most public training sets), and bingbot, which matters more than it used to because Copilot and a chunk of other assistants sit on Bing's index.

Bytespider is missing from that list deliberately. ByteDance publishes no official documentation for it, so we will not tell you it behaves in a way we cannot verify.

The rule that usually makes sense

If you want to be found in AI answers but would rather not be training data:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Two caveats before you paste that in. First, robots.txt is a request, not a lock — well-behaved operators honour it and the rest never asked permission. Second, the training/retrieval line is blurrier than the operators' documentation makes it sound, and it may move. If your position is "never use my work for training", robots.txt is a polite start, not an enforcement mechanism.

Two things that break this file silently

Group precedence. Under RFC 9309 a crawler obeys the most specific group matching its name, and only that group. If you have User-agent: * with Disallow: /private/ and a separate User-agent: GPTBot group, GPTBot ignores the /private/ rule entirely. Every rule you want it to follow has to be repeated in its own group. This is the single most common bug we find in real robots files.

A stray Disallow: /. Usually from a staging config that shipped, or a CMS plugin that "helpfully" rewrote the file. It blocks everything, and nothing in your analytics will tell you.

Check yours

The checker below reads your live robots.txt, resolves each agent against the actual RFC 9309 matching rules rather than a substring search, and tells you which of the sixteen are allowed and which are not. No sign-up, and it runs in this page.

Once you know what your file says, the interesting question is what the crawlers are doing with that permission — which pages they fetch, how often, and whether the thing claiming to be GPTBot in your access log is actually coming from OpenAI's published IP ranges. A worrying share of it is not.