GPTBot

OpenAI's crawler for model training data. Guides often conflate it with two other OpenAI user agents that do different jobs. This page keeps them separate, in OpenAI's own words. Source for every quote: OpenAI's bot documentation, read 13 September 2026.

Three OpenAI user agents, three different jobs

Token	What OpenAI says it's for	Disallowing it means
`GPTBot`	"Used to crawl content that may be used in training our generative AI foundation models."	"Indicates a site's content should not be used in training generative AI foundation models."
`OAI-SearchBot`	"Used to surface websites in search results in ChatGPT's search features."	Your site "will not be shown in ChatGPT search answers, though can still appear as navigational links." OpenAI notes "it can take ~24 hours from a site's robots.txt update for our systems to adjust."
`ChatGPT-User`	"When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent." It "is not used for crawling the web in an automatic fashion."	OpenAI: "Because these actions are initiated by a user, robots.txt rules may not apply." A Disallow records your preference; it is not documented to stop these fetches, and it has no effect on search: "ChatGPT-User is not used to determine whether content may appear in Search."

OpenAI's robots.txt controls are GPTBot and OAI-SearchBot, and they are independent: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models". Blocking training while staying in search is a coherent, common choice, and our scanner never counts a training opt-out against you (how the two robots checks score). One detail worth knowing: "if your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicative crawling". OpenAI also documents a fourth agent, OAI-AdsBot, which "only visits pages submitted as ads", which is not something a robots.txt rule on an ordinary site needs to address.

Block only GPTBot (training), allow search

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Block training and search

User-agent: GPTBot
User-agent: OAI-SearchBot
Disallow: /

Adding a User-agent: ChatGPT-User line to that group is harmless and states your preference, but per OpenAI's documentation it may not prevent a fetch a user asked for. Our scanner reads a ChatGPT-User rule as declared policy and scores it with the other retrieval agents; it cannot observe whether OpenAI applies it.

Allow both

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

If you have no group for a token at all, your default (User-agent: *) group applies to it: a named group is used when one exists, and * only when none does. That is the standard rule (RFC 9309 §2.2.1), and how our scanner evaluates your file.

Consequences either way

A robots.txt rule is a request to future crawls; nothing on OpenAI's page says a new Disallow removes content from a training run that already happened, or affects data obtained by other means. Allowing GPTBot costs you nothing in server load beyond ordinary crawler traffic, and doesn't obligate you to anything.

A directive in robots.txt only works if the crawler reads and honors it. It says nothing about whether your edge (a WAF, a CDN bot-management setting) is separately blocking (or failing to block) self-identifying crawlers as a category regardless of what robots.txt says. That's a different failure mode. See how we test for it with our own crawler, never by sending GPTBot's user agent, so a block is strong evidence and a pass is weaker.

Check your site's current policy →