Technical access determines whether a crawler or retrieval tool can request your pages. Check the HTTP response and returned content for the agents relevant to your audience. A page that works in a browser can still fail for an automated request.
Document crawler access
Separate training crawlers, search indexers, and user-triggered fetchers. Document the purpose of each access rule. Crawlers, Indexes, and Training Data explains the distinctions.
Providers publish crawler names and access policies. Use those documents when configuring robots.txt or bot-management rules. A user-agent name identifies a claimed client; verify supported IP ranges or other provider verification methods before treating a request as authentic.OpenAI's bot documentation lists crawler purposes and published IP ranges for verification.Anthropic's crawler documentation describes each agent and what disabling it costs you.Perplexity's crawler documentation, including the published IP list at perplexitybot.json.Google's crawler and user-agent list documents Google-Extended and how it differs from search crawling.
Decide which content should be available for each documented use. A training-crawler rule and a search-access rule can have different effects. Avoid promising that an allow rule produces inclusion or that a block removes every mention of the business.RFC 9309, the Robots Exclusion Protocol standard: how groups, user-agent matching, and allow/disallow precedence actually work.
Rendering: what an AI fetcher actually sees
Check the initial HTML for essential descriptions, prices, links, and conditions. Then check the rendered page. Server-rendered text reduces dependence on JavaScript support, but a local HTML check does not reproduce every provider’s renderer.Google's AI features guide confirms JavaScript processing for Google's pipeline while pointing to its JavaScript SEO best practices.
Measure status codes, response time, redirects, and returned content. A curl request with a bot user agent can identify some problems. It does not prove what a real provider receives from another IP address or what its extraction system retains.
Access errors and challenges
A CDN, web application firewall, or bot-management rule can return an error or challenge to a crawler while the page remains usable by a person. Check those services alongside the application.
Look for 403, 429, and challenge responses in access logs. Review the rules responsible before changing them. A failed fetch can prevent that request from obtaining page content, but its effect on an answer depends on other available sources.
Log-file analysis: verifying who fetches what
Server logs show requests that reached the server and the responses it sent. They do not show every operation in an AI product. A verified fetch establishes access, not that the page was cited or recommended.
Use this checklist for the selected pages:
- Access decisions are documented for the relevant agents and uses.
- CDN, firewall, and bot-management responses have been checked.
- Important pages return the expected content in the tested HTML and rendering conditions.
- Response times and rate-limit behavior have been reviewed for verified requests.
- Canonical URLs, indexing, and snippet controls have been checked for the relevant search provider.
- Logs identify available agents, pages, status codes, and verification details.
Treat llms.txt as a separate proposal. Check whether the specific product you use documents support before relying on it. See Structured Data and Machine-Readable Facts.
Google’s separate AI inclusion control
As checked on September 10, 2026, Search Console has a Search generative AI setting for inclusion in AI Overviews, AI Mode, and generative features in Discover. Check the effective setting and any inherited parent-property setting during an audit.Google Search generative AI control. Google states that this control completed worldwide rollout on August 31, 2026. It is separate from training controls.
Keep the decisions separate: Googlebot access, indexing and preview directives, Search generative AI inclusion, and Google-Extended. An audit that checks only robots.txt can miss the property setting. Record the current choice before proposing a change.
A small access test
Choose the home page, one service page, and the pricing page. Request each URL and save the response headers and body. For each, record the final URL, status, visible title, main text, and whether the required facts are present.
Then compare those results with an available provider inspection tool or verified crawler request. If the browser succeeds but the automated request receives a challenge page, record the exact rule and request evidence. A successful request with a copied user-agent string is only a local test.