ChatGPT disclosed things in this session that I did not expect it to disclose, and there is no guarantee it stays available. I would rather people had time to use it than had a tidier version of it later.
So: every word of every ChatGPT response here is verbatim, and that part is checked automatically on every build. What is not finished is the presentation. The color coding on the code blocks is incomplete and some of it is imprecise, and there are notes to myself still sitting in the page.
Parts 1 and 2 are close to empty and are getting a lot of detail over the next week or so, along with a cleanup pass on everything else. Worth checking back.
Fan-out search
How Fan-Out Search Changes AI Recommendations
The same test with search switched on, where the irrelevant word ends up inside the search queries.
This section is not written yet. The research behind it is finished and the numbers below are verified, but the writing and the exhibits are still to come.
Same test as Part 1, with search switched on. This is the bridge into the rest of the resource, because it shows the prompt turning into actual search queries before anything gets fetched.
What it shows
- Turning search on only offers the tool. Across six models the rate at which they actually searched when allowed to ranged from 10% to 100%.
- The car ends up in the queries. Across 1,230 captured search queries, luxury vocabulary appears in 18% to 23% of Ferrari and BMW queries and 0% to 4% of Civic queries. Budget vocabulary only ever appears for the Civic.
- The car's own name leaks in. Up to 6% of queries name the vehicle directly, in strings like "best luxury hotels Los Angeles secure valet parking Ferrari Beverly Hills official".
- Fan-out width varies enormously. One model issues about 11 queries per call. Another issues exactly one. On models with a reasoning dial, width scales with it, from under 3 queries at the lowest setting to nearly 17 at the highest.
- Searching changes the answer. Memory answers and searched answers disagree more than two runs of the same setting do, for five of the six models.
The query tables are the exhibit that matters here. They show an irrelevant detail being translated into a different question before the machine goes looking for anything. Part 4 picks up from exactly that point and shows what those queries return.
Full write-up to come.