ChatGPT disclosed things in this session that I did not expect it to disclose, and there is no guarantee it stays available. I would rather people had time to use it than had a tidier version of it later.
So: every word of every ChatGPT response here is verbatim, and that part is checked automatically on every build. What is not finished is the presentation. The color coding on the code blocks is incomplete and some of it is imprecise, and there are notes to myself still sitting in the page.
Parts 1 and 2 are close to empty and are getting a lot of detail over the next week or so, along with a cleanup pass on everything else. Worth checking back.
AI Search Optimization
How AI Search Works, From Prompt to Response
A guide for SEOs, AI search marketers, and marketing teams
This section is not written yet. The research behind it is finished and the numbers below are verified, but the writing and the exhibits are still to come.
Same test as Part 1, with search switched on. This is the bridge into the rest of the resource, because it shows the prompt turning into actual search queries before anything gets fetched.
What it shows
- Turning search on only offers the tool. Across six models the rate at which they actually searched when allowed to ranged from 10% to 100%.
- The car ends up in the queries. Across 1,230 captured search queries, luxury vocabulary appears in 18% to 23% of Ferrari and BMW queries and 0% to 4% of Civic queries. Budget vocabulary only ever appears for the Civic.
- The car's own name leaks in. Up to 6% of queries name the vehicle directly, in strings like "best luxury hotels Los Angeles secure valet parking Ferrari Beverly Hills official".
- Fan-out width varies enormously. One model issues about 11 queries per call. Another issues exactly one. On models with a reasoning dial, width scales with it, from under 3 queries at the lowest setting to nearly 17 at the highest.
- Searching changes the answer. Memory answers and searched answers disagree more than two runs of the same setting do, for five of the six models.
The query tables are the exhibit that matters here. They show an irrelevant detail being translated into a different question before the machine goes looking for anything. Part 4 picks up from exactly that point and shows what those queries return.
Full write-up to come.
Look up any permutation yourself
The same explorer as Part 1, with the web-search dropdown added and set to on. Flip it to off and you are back on Part 1's memory-only route with the same model and setting selected, which is the cheapest way to see how much of an answer search rewrites.
Pick a model, a dial setting, and whether web search was enabled. The tables show everything that exact slice of the 1,363 usable answers recommended: how often each site made the top 10, and how often it was the #1 pick. The query tables underneath show what the models actually typed into the web for the same selection.
Car against car: both wordings of each combined
Each car pooled across both of its wordings ("a Civic" with "a used Honda Civic", and so on), so the comparison is car against car with the phrasing averaged out. If the car itself did not matter, these three columns would hold the same list. In the grid there is one row per domain, in the order the columns introduce them: the first column's top 10, then whatever the second column adds, then whatever the third adds. Each cell is that domain's rank for that column, and a blank cell means it did not make that top 10 at all. Teal is a top-3 finish, blue is 4th to 7th, purple is 8th to 10th, and the shading runs dark at rank 1 to light at rank 10 straight through, so only the hue changes at a band edge.
What the models typed into the web, same selection
Every web search the models issued for this selection, counted rather than ranked, most-used first. These exist only for search-enabled calls, so the tables empty out if you set the web-search dropdown to off. Repeats above 1 mean separate calls independently invented the same query. Watch the vocabulary move across the three columns: the same request becomes a different question asked of the web.
One word different: "a Civic" / "a BMW" / "a Ferrari"
The three one-word wordings on their own. Everything reads the same way as the grid above.
What the models typed into the web, same selection
The same three cars, described
The three descriptive wordings, rows ordered by "a used Honda Civic" first, then the newcomers from "an off-lease BMW 3 Series" and "a brand new Ferrari".
What the models typed into the web, same selection
The same thing again, by hotel name instead of website
We asked for websites, so that is what the models gave us, and they spelled the same hotel three different ways: a plain address like westin.com, a page inside the parent company like marriott.com/en-us/brands/westin, or a subdomain like westin.marriott.com. Everything above treats the last two as Marriott, which quietly merges brands that compete with each other. Here is the same data with every address turned into the hotel it actually points at. The rule is one row per brand, so a named sub-brand always stands on its own, while a page for one location folds into the brand that runs it. The dropdowns above drive these tables too.
Car against car, by hotel name
One word different, by hotel name
The same three cars described, by hotel name
Names come from a researched lookup covering 99.7% of the 13,646 addresses the models returned. The rest are shown as the raw address, and almost all of those are domains the models invented, like parkerhyatt.com or thepenisula.com. Those are left exactly as the model wrote them: a made-up address is a real result, and repairing it would hide it.
"In top-10" is the share of the selection's answers that contain the entry anywhere; "#1" is the share where it was the first recommendation. Rows are ordered by how often the entry appears, position-weighted on ties. Domains are rolled up to the registrable domain, and n under each wording is the number of usable answers in the current selection. Each table scrolls; the full list continues past the first ten rows.