ClaudeBot

Anthropic's crawler for model training data. Anthropic runs three bots for three different jobs and documents each separately, so treat them separately in robots.txt. Source for every quote: Anthropic's support article on web crawling, read 13 September 2026.

Token	What Anthropic says it's for	Disallowing it
`ClaudeBot`	"Collecting web content that could potentially contribute to their training."	"Signals that the site's future materials should be excluded from our AI model training datasets."
`Claude-User`	"When individuals ask questions to Claude, it may access websites using a Claude-User agent."	"Prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search."
`Claude-SearchBot`	"Navigates the web to improve search result quality for users."	"Prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results."

Anthropic states that its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt" and "respect anti-circumvention technologies", and, unlike OpenAI's and Perplexity's user-triggered agents, it makes no exception for Claude-User. Two practical details from the same page: opt out "for every subdomain that you wish to opt out from" (a rule on example.com does not cover docs.example.com), and Anthropic "support[s] the non-standard Crawl-delay extension" if you want to slow it rather than block it. Anthropic also says that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out" because it prevents the bot from reading your robots.txt, which is the documented mechanism.

Anthropic's documentation names only these three. A fourth token, anthropic-ai, appears in many robots.txt files as an Anthropic crawler; Anthropic's page does not mention it, so blocking it has no documented effect. See the reference table for how our scanner treats it.

Block only ClaudeBot (training), allow the rest

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

Our scanner never counts a training opt-out against a site; it does count a disallowed Claude-User or Claude-SearchBot as a visibility decision (how the two robots checks score). If you block ClaudeBot, block anthropic-ai in the same edit: leaving that token allowed is the one training-policy gap the scanner flags.

Block all three Anthropic bots

User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
Disallow: /

Add User-agent: anthropic-ai to the same group if you want that token covered too; it costs nothing.

Consequences either way

As with any crawler, a robots.txt rule is a request to future crawls. Nothing on Anthropic's page says a new Disallow removes content already collected, and it doesn't prove or disprove anything about how your edge (Cloudflare, a WAF, a security plugin) treats self-identifying crawlers independently of your stated policy. That mismatch is what our crawler access test looks for, with our own crawler, never by sending ClaudeBot's user agent, so a block is strong evidence and a pass is weaker.

Check your site's current policy →