Where the Machines Get Their Data: Crawlers, Indexes, and Training Pipelines

Three separate supply chains feed AI search, and they run on three different clocks. Training corpora are assembled from web crawls and licensed data, then frozen into a model for months or years. Live search indexes, Google's, Bing's, and Brave's among them, refresh continuously and feed the retrieval step. And dedicated fetchers hit your pages at answer time, seconds after a user asks. Every bot in your server logs belongs to one of these pipelines, every robots.txt decision affects exactly one of them, and mixing them up is how sites end up blocking the wrong thing for the right reason.

Three pathways, three timelines

Training ingestion collects text that may enter a future model's weights. Whatever a training crawler takes today shows up, compressed into statistical associations, in a model that ships months from now and then serves answers for years; How an LLM Turns a Prompt into a Response covers how that compression works. Index retrieval is the familiar middle path: ordinary search crawling, feeding the indexes that AI products query when they run the fan-out searches described in Retrieval and Grounding: How an AI Answer Gets Assembled. Answer-time fetching is the newest path: a user asks about your product, and a fetcher requests your page right then, on behalf of that one conversation.

The pathways age differently. A page published this week can enter answers through retrieval within days, while the same page will not influence any model's memory until a training run picks it up and the resulting model ships.

The bot roster

The major operators split their crawling by purpose and document each user agent separately.The rows below come from each operator's own documentation: OpenAI's bot docs, Anthropic's crawler docs, Perplexity's crawler docs, Google's crawler list, and Common Crawl's CCBot page.

Google-Extended is the odd one out: no crawler carries that name. Google crawls with its existing user agents, and Google-Extended is a robots.txt control token deciding whether the crawled content may be used to train future Gemini models and for grounding, Google's term for feeding Search-index content to Gemini at prompt time. Google states that the token affects neither a site's inclusion in Google Search nor its ranking.Google's crawler documentation: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

What blocking each bot actually costs

Each operator treats its tokens independently: OpenAI documents that a site can allow OAI-SearchBot, to stay in ChatGPT search results, while disallowing GPTBot, to opt out of training.OpenAI's bot documentation, which also notes that robots.txt changes take roughly 24 hours to reach its systems, and that sites opted out of OAI-SearchBot stop appearing in ChatGPT search answers. So the decision is per pathway, and the costs differ by pathway. Blocking a search bot cuts current visibility: your pages stop entering that product's answers within days. Blocking a training bot changes nothing you can see now; the cost lands quietly, in the memory of every future model trained without your content.

This clock difference explains a confusion that shows up constantly: "we blocked GPTBot" and "ChatGPT still talks about us" are both true. The models currently serving answers were trained on crawls collected before the block, third parties keep writing about you regardless, your site persists in older Common Crawl archives, and licensed datasets are untouched by your robots.txt. The block only starves future training runs of your first-party pages. The reverse holds too: blocking today echoes for years, because a model trained without your content keeps answering from that gap until a later model replaces it. That persistence is why this is the most consequential robots.txt era since the file was invented, and why the default this book recommends, worked through in Technical Accessibility for AI Crawlers, is to allow deliberately rather than block reflexively.

Two facts should discipline the decision. First, robots.txt is a published preference, formalized as RFC 9309, and compliance is voluntary.RFC 9309, the Robots Exclusion Protocol; the standard defines how compliant crawlers behave and carries no enforcement mechanism. Cloudflare has reported Perplexity responding to blocks with an undeclared crawler impersonating Chrome on macOS, rotating IPs and ASNs, and sometimes not fetching robots.txt at all; Common Crawl likewise warns of impostor bots claiming to be CCBot. Second, AI crawling is heavy relative to what it sends back: Cloudflare measured crawl-to-refer ratios in June 2025 ranging from roughly 70,900 Anthropic page requests per referral down to Mistral's 0.1, with the caveat that app traffic carries no referrer and inflates the ratios.Cloudflare's crawl-to-refer analysis on Radar, June 19-26, 2025. The ratios move constantly; the direction, more crawls per referral over time, has been steady. Weigh server load and your content-licensing bargaining position against visibility in every model trained from now on, and make the call per bot, on purpose.

Licensing: the paid lane into training data

Alongside crawling, the operators buy data. Reddit licenses its corpus to Google for an estimated $60 million per year and has a parallel deal with OpenAI.Columbia Journalism Review's analysis of Reddit's licensing position, which attributes the $60 million per year estimate to María Garcia of Implicator.ai. News publishers have struck similar deals across the industry. For visibility work, the mechanism matters more than the sums: licensed corpora enter training runs on contract terms, robots.txt does not apply to them, and a publisher inside one holds a durable position in model memory that outlives any single news cycle. Earning coverage on the platforms and publishers that models ingest wholesale is part of the case made in Digital PR, News, and the Gatekeeper Publishers and Reviews, Reddit, and Communities.