An AI answer uses learned model parameters and the context provided for the request. That context may include the prompt, conversation history, uploaded material, and retrieved sources. This chapter compares answers with web retrieval against answers without it.
The comparison helps identify changes associated with search access. It cannot assign every sentence or brand mention to a single source.
The memory answer
Without retrieval, a model still receives the prompt and other supplied context. Its learned parameters affect how it responds. “Memory-only” in this guide refers to a test without external retrieval, not an answer generated without any input.
A deployed model does not automatically learn a website change. Its published cutoff also does not guarantee knowledge of every earlier fact. Check current business information explicitly in generated answers.OpenAI publishes each model's knowledge cutoff in its model documentation; the other providers list theirs in model cards. A cutoff is a model-specific date, not a guarantee that every earlier fact is known.
Public coverage and brand prominence are associated with recommendations in the OppAlerts study. Output tests cannot directly measure a brand’s representation in private training data. See the correlation analysis.
Repeated requests can produce different brand lists. Compare batches of responses under consistent conditions rather than a single answer from each mode.
What search changes
With search available, a product can retrieve information for the answer. Record whether search actually ran when the interface exposes that information. A supported tool setting and a completed tool call are different observations.
Compare the brands in each condition. Count brands present in both sets and those present in only one. Also compare factual accuracy. Search can supply current information even when the list of recommended businesses stays the same.
For example, suppose an answer without search names A, B, and C. A searched answer names B, C, and D. Two of the four distinct brands appear in both answers. Their Jaccard overlap is 2 / 4, or 50%. This is an invented example of the calculation, not a result from the OppAlerts study.
Repeat both conditions before describing that difference as typical. The earlier draft’s aggregate percentages are withheld until their matching source tables are available.
Recommendations without an own-site fetch
Check whether the available logs show a request to the recommended business’s domain. Record that separately from references to the business on third-party pages.
A model could use a review, directory, news article, or search extract without fetching the business’s site. An absent own-site request therefore cannot establish that the recommendation came only from learned parameters.
When the interface does not expose requests, use “not observable” rather than “no retrieval.” A later explanation written by the model is not a substitute for the original tool record.
Planning changes you can verify
Changes to learned parameters require a model update or another intervention by the provider. A publisher cannot schedule that process or guarantee that new coverage will be included.
Retrieval can make current pages available without a model update. The delay depends on crawling, indexing, access, and source selection.
Plan against observable work: correct information, make pages accessible, and measure responses after changes. Treat future training effects as uncertain.
| Work | Directly verifiable result | AI outcome to measure |
|---|---|---|
| Technical access | Expected page content returned | Source availability in observed searches |
| Content and factual edits | Accurate, readable pages published | Accuracy and citations in repeated answers |
| Structured data and feeds | Valid, consistent supported data | Relevant product inclusion where observable |
| Business profiles | Correct facts on maintained profiles | Fewer observed identity errors |
| Independent coverage | Relevant, accurate material published | Mentions and citations, with causal limits |
| Community participation | Questions answered and problems addressed | Observed use of discussion sources |
| Paid placement | Verified campaign delivery | Ad metrics and conversion results |
The table groups work by its possible route of influence. Paid placement is measured separately in Ads in AI Search.
Prioritize changes you can verify directly, such as corrected pricing or accessible pages. Evaluate independent coverage for its audience and business value as well as any observed AI visibility change.
Seeing the split yourself
Choose a small set of representative prompts. Run each repeatedly with search disabled and enabled where the interface supports both. Keep other settings consistent and compare mention rates, citations, and accuracy. More runs may be needed when the difference is small.
The full measurement method is in Measuring AI Search Visibility.