AI products can obtain information through training datasets, search indexes, and requests made during a user session. These uses have different controls. Identify the documented purpose of a crawler before changing access.
Three uses of collected content
Training collection gathers material that may be used in a future model. Search crawling collects pages for an index. A user-triggered fetch requests content for a particular task. A provider may document separate agents or policy controls for these uses.
A newly published page may become available through retrieval before it could affect a future model. Neither indexing nor inclusion in training is guaranteed. Avoid promising a fixed time from publication to an AI recommendation.
Crawler purposes
The table lists documented crawler purposes. Verify the current provider documentation before applying a rule.The rows below come from each operator's own documentation: OpenAI's bot docs, Anthropic's crawler docs, Perplexity's crawler docs, Google's crawler list, and Common Crawl's CCBot page.
| User agent | Operator | Feeds |
|---|---|---|
| GPTBot | OpenAI | Training: content that may be used to train foundation models |
| OAI-SearchBot | OpenAI | Index: ChatGPT's search features |
| OAI-AdsBot | OpenAI | Checks submitted advertising landing pages; not foundation-model training |
| ChatGPT-User | OpenAI | Answer-time: page visits triggered by user requests |
| ClaudeBot | Anthropic | Training: content that may contribute to model training |
| Claude-SearchBot | Anthropic | Index: search result quality for Claude |
| Claude-User | Anthropic | Answer-time: page visits triggered by user questions |
| Google-Extended | Training and grounding for Gemini (a robots.txt token, described below) | |
| PerplexityBot | Perplexity | Index: Perplexity search results, per Perplexity not model training |
| Perplexity-User | Perplexity | Answer-time: page visits triggered by user questions |
| CCBot | Common Crawl | Public web collection that may be used in datasets |
Google-Extended is a robots.txt control token rather than a separate crawler user agent. Google documents its scope separately from Google Search. Do not substitute a Google-Extended rule for the Googlebot controls that govern Search.Google's crawler documentation: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
Choosing crawler access
OpenAI documents separate controls for GPTBot and OAI-SearchBot. A site can make different choices about training collection and search access. The effect of a block depends on the agent and provider; it is not a guarantee that every mention or link disappears.OpenAI's bot documentation, which also notes that robots.txt changes take roughly 24 hours to reach its systems, and that sites opted out of OAI-SearchBot stop appearing in ChatGPT search answers.
A crawler block does not remove information already learned by a deployed model. An answer can also use third-party sources or information supplied by the user. Record the purpose of each block and verify its behavior in server logs.
Robots.txt specifies rules for cooperating crawlers. It is not access authentication. Consider server load, licensing policy, and intended discoverability separately when deciding which requests to allow.RFC 9309, the Robots Exclusion Protocol; the standard defines how compliant crawlers behave and carries no enforcement mechanism. Cloudflare has reported Perplexity responding to blocks with an undeclared crawler impersonating Chrome on macOS, rotating IPs and ASNs, and sometimes not fetching robots.txt at all; Common Crawl likewise warns of impostor bots claiming to be CCBot.Cloudflare's crawl-to-refer analysis on Radar, June 19-26, 2025. The ratios move constantly; the direction, more crawls per referral over time, has been steady.
Licensed data
Providers may also obtain data through licensing agreements. Public announcements can establish that an agreement exists, but not how an individual article affects recommendations. Treat relevant coverage as a communication and discovery activity with uncertain model-training effects.Columbia Journalism Review's analysis of Reddit's licensing position, which attributes the $60 million per year estimate to María Garcia of Implicator.ai.
Policies for user-triggered requests
Do not assume every provider treats user-triggered retrieval the same way. OpenAI says robots.txt may not apply to ChatGPT-User requests. Perplexity says its user fetcher generally ignores robots.txt. Anthropic says its listed bots honor robots.txt.OpenAI crawler documentation. Separates search, training, advertising checks, and user-triggered requests.Perplexity crawler documentation. States the policy for Perplexity-User and provides verification information.Anthropic crawler documentation. Describes robots.txt behavior for ClaudeBot, Claude-User, and Claude-SearchBot.
For Google, also check the Search Console AI inclusion setting. A training preference, a search-indexing decision, and an answer-time access decision need separate records.