AI Crawlers: The Reference

The 13 AI agent tokens our scanner reads from your robots.txt: who runs each one, what the operator says it is for, whether it is a real HTTP crawler or only a permission flag, and what disallowing it does, plus the other agents the same operators document, which our scanner does not read from your file. Two of the thirteen are user-triggered fetchers (ChatGPT-User, Claude-User): what we score is the policy you declared for them, not whether the operator applies it. OpenAI says robots.txt "may not apply" to ChatGPT-User, Anthropic makes no such exception. Every row is sourced to the operator's own documentation, read on the review date. Last reviewed: 13 September 2026.

This table describes the robots.txt tokens each operator documents. We verify robots.txt syntax and precedence when we scan a site (how the two robots checks are scored). We do not verify that any operator's live crawler actually obeys the rule in the wild: that would require watching their traffic, not yours, and we have no way to do that from outside. Where an operator's documentation does not itself say the crawler follows robots.txt, the row says so.

Retrieval agents: the five our scanner scores for visibility

These fetch pages so an assistant can find, cite or link them. Disallowing one is a visibility decision; our crawl.robots_retrieval check counts it.

Training tokens: the eight our scanner reads for policy coherence

Content fetched under these is used to build models; with one documented exception (Google-Extended, which also governs grounding in Gemini apps), nothing is documented to come back to your page as a citation or a visit. Disallowing them is a rights decision our scanner never penalises: the crawl.robots_training check only looks for a same-operator gap. Two of the eight are not crawlers at all, and one is documented by its operator as a search crawler, not a training one; the table says which.

"Robots-only token" means no crawler ever shows up in your logs carrying that exact string; it is a permission flag the operator's existing crawlers check internally (Google's, and Apple's Applebot), not a separate bot that fetches your pages. Every other row above is a real HTTP crawler or user agent: something you can watch arrive in your access logs with that product token in its User-Agent. Operators warn that these strings are spoofed; OpenAI, Anthropic, Perplexity, Common Crawl, Apple and ByteDance each publish IP ranges or a reverse-DNS pattern for checking.

Documented, not scored: user-triggered fetchers and others

The same operators document tokens our scanner does not read, because their operators say robots.txt may not apply or because they are not about AI at all. If you see one in your logs, this is what it is.

How to allow or block one

Robots.txt rules are grouped by User-agent block. To allow everything:

User-agent: GPTBot
Allow: /

To block one crawler while leaving your default group untouched:

User-agent: GPTBot
Disallow: /

To block several in one place, list each token in its own group, or put several User-agent lines on one group. Both are valid, and the second is what most sites actually do:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

A token with no group of its own falls back to your User-agent: * group: every group that names a token applies to it, and * is used only when none does. Our scanner evaluates your file the same way, at the path of the page you scan (how). Agent-specific pages: GPTBot, ClaudeBot, PerplexityBot.

Directive syntax isn't the same as edge enforcement

A correct Disallow: / line only stops crawlers that choose to read and honor it. It does nothing to a WAF rule, a Cloudflare bot-management setting, or a CDN default that blocks a user agent before your robots.txt is ever fetched, and it does nothing to the reverse case, where your policy allows crawlers but your edge quietly turns self-identifying crawlers away anyway. That mismatch is exactly what a robots.txt parser alone can never show you, and it's the finding our own scanner is built around, observed with our own crawler only, so a block is strong evidence and a pass is weaker. See how we test it, or run a free scan below.

Run a free scan of your site →

Tokens and vendor documentation drift over time; the review date above is when each source was last read. If you notice one of these rows is out of date, tell us and we'll fix it. See also: llms.txt, explained honestly and how our own crawler identifies itself.