If AI bots cannot fetch and parse your pages, nothing else in this book functions. Your writing never enters an answer, your markup never gets read, your evidence never gets cited. This chapter is the plumbing check: which bots you let in, what they can actually see when they arrive, and the configuration mistakes that lock them out without anyone noticing.
The access matrix: every bot, an explicit decision
AI bots come in three classes, and each class deserves its own decision. Training crawlers collect pages for future model training. Search indexers build the index an AI product retrieves from at answer time. User-triggered fetchers grab a specific page because a user's question needs it right now. Where the Machines Get Their Data: Crawlers, Indexes, and Training Pipelines covers what each pipeline does with your content; this chapter covers letting the right ones through.
The major providers document their agents and treat each setting as independent. OpenAI runs GPTBot for training, OAI-SearchBot for search indexing, and ChatGPT-User for user-triggered fetches, and states explicitly that you can allow one while blocking another.OpenAI's bot documentation lists all three agents, their purposes, and published IP ranges for verification. Anthropic runs ClaudeBot for training collection, Claude-SearchBot for search, and Claude-User for user-triggered access.Anthropic's crawler documentation describes each agent and what disabling it costs you. Perplexity runs PerplexityBot for its search index and Perplexity-User for user requests, and states that neither collects content for training foundation models; robots.txt changes take up to 24 hours to register.Perplexity's crawler documentation, including the published IP list at perplexitybot.json. Google folds AI training control into the Google-Extended product token, separate from Googlebot itself, so you can stay in Google Search while opting out of Gemini training use.Google's crawler and user-agent list documents Google-Extended and how it differs from search crawling.
Express the decisions in robots.txt, which supports per-agent rule groups under RFC 9309.RFC 9309, the Robots Exclusion Protocol standard: how groups, user-agent matching, and allow/disallow precedence actually work. For a brand that wants AI visibility, the default is allow all three classes. Blocking a search indexer or a user-triggered fetcher removes you from answers directly. Blocking a training crawler is a real strategic choice with a delayed cost, next year's model memory, and the tradeoff argument lives in the crawlers chapter. Whatever you choose, choose it per bot, on purpose, and write it down.
Rendering: what an AI fetcher actually sees
Assume an AI fetcher sees your raw server HTML and nothing more. Google documents that it can process JavaScript-rendered content when it is not blocked.Google's AI features guide confirms JavaScript processing for Google's pipeline while pointing to its JavaScript SEO best practices. The other providers do not publish rendering capabilities for their fetchers, and observed behavior across the industry points to little or no JavaScript execution, so treat client-side rendering as content the non-Google half of the ecosystem cannot see. That assessment would change if providers document real rendering; until then, server-rendered HTML is the safe default for every page you want in answers.
Answer-time fetches also run on a clock. When ChatGPT-User or Perplexity-User hits your page mid-answer, the page has seconds to return parseable text before the system moves on and builds the answer from whoever did respond. Slow origins, interstitials, and cookie walls all spend that budget. Test it yourself: request your key pages with each bot's user-agent string via curl, time the response, and read what comes back as text. What you see in that output is your actual product page, as far as AI search is concerned.
Silent blockers: the 403 nobody sees
The most common technical failure in AI search is a CDN, WAF (web application firewall), or bot-management layer shipping with "block AI bots" defaults that the marketing team never hears about. The site works perfectly in a browser, every human check passes, and every AI fetch dies with a 403. Nothing on the surface looks broken, which is why this failure survives for months.
Work through it methodically. Pull the WAF and CDN logs and search for the user agents above with 403, 429, or challenge responses. Check the bot-management console for AI-bot category rules that someone enabled by default. Then verify direction by fetching your pages from outside your own network with the bot user-agent strings. Rate limits deserve the same scrutiny as blocks, because an answer-time fetcher gets one shot; a 429 at the wrong moment costs the citation just as surely as a 403.
Log-file analysis: verifying who fetches what
Server logs are the ground truth for all of the above. Filter them by the documented user agents, then verify the traffic is genuine against the providers' published IP ranges, since anyone can put GPTBot in a user-agent header. From there, three questions run the whole diagnostic: which bots fetch you at all, which pages they fetch, and what status codes and response times they get. Fetch counts by user-triggered agents are also a leading indicator that your pages are entering real answers, before referral traffic shows it; Attribution and Business Impact picks that thread up.
The access checklist, condensed:
- robots.txt has an explicit rule for every documented AI agent, per class: training, search, user-triggered.
- CDN, WAF, and bot-management rules audited for AI-bot blocks and challenges.
- Key pages return full content as server-rendered HTML, verified with bot user-agent fetches.
- Response times on key pages hold under load, and rate limits exempt verified AI agents.
- Canonicals and indexability are clean, since AI search products retrieve from search indexes; a page Google cannot index is a page most AI answers cannot use.
- Logs reviewed monthly: agents, pages, status codes, verified IPs.
One file you may hear about here is llms.txt, a proposed index file for LLMs. It is experimental, no major platform documents consuming it, and it belongs to the discussion in Structured Data and Machine-Readable Facts.