OA9 OppAlerts from Ben Wills
How AI Search Works: From Prompt to Response A guide for SEOs, AI search marketers, and marketing teams
Work with me
Work in progress Work in progress. Released about a week early, on purpose.

ChatGPT disclosed things in this session that I did not expect it to disclose, and there is no guarantee it stays available. I would rather people had time to use it than had a tidier version of it later.

So: every word of every ChatGPT response here is verbatim, and that part is checked automatically on every build. What is not finished is the presentation. The color coding on the code blocks is incomplete and some of it is imprecise, and there are notes to myself still sitting in the page.

Parts 1 and 2 are close to empty and are getting a lot of detail over the next week or so, along with a cleanup pass on everything else. Worth checking back.

Read the announcement →

Evidence and sourcing

Which Parts of This Session Are Real

Every code block is colored by where its values came from. Read this before the transcript.

The transcript in Part 4 is full of code blocks. Some of them are the real thing. Some of them ChatGPT made up. They look almost identical, and telling them apart is the whole point.

So every code block is colored.

green
Real. The values came from an actual call in that session. This covers the exact arguments ChatGPT sent to a tool, the exact text that came back, and ChatGPT's own JSON when the values in it are real.
blue
Invented. ChatGPT made the values up. The shape may be about right. The numbers and strings are not from anywhere.
amber
Documented. Copied from OpenAI's published documentation. You can go and check this yourself, which makes it the most verifiable material here.
purple
Conceptual. Proposed, inferred, or a shape shown only to say it never arrives. Also the drawn diagrams, which are ChatGPT's picture of its own process rather than real functions.
unmarked
Not yet sorted. The rules could not place it with confidence, so it carries no color and says so on its own label. A wrong green would be worse than no answer at all. About a third of the blocks are still in this state, most of them in the worked examples.
no color
Not a representation. Code that is not a tool payload at all, such as the reflection prompt itself.

The labels underneath

Each colored block also carries a short label saying what it is. Green splits into four of them, because "real" covers four different things.

LabelWhat it means
sent exactly as shownThe arguments that went to the tool
returned exactly as shownThe text that came back from the tool
ChatGPT JSON built from real valuesIts own JSON, holding values a call returned
ChatGPT JSON, invented valuesIts own JSON, holding made-up values
field names only, values are typesA list of which fields exist, with type words where values would sit
OpenAI documentationFrom the published API docs
not exposed or proposedA shape shown to say it never arrives, or one suggested for a system that does not exist
conceptual, drawn by ChatGPTA diagram of the process, not real functions
unclassified, needs reviewNot sorted yet

Where green stops and blue starts

There is a specific moment in the transcript where the colors flip, and it is worth knowing about before you read.

For the first seven turns, ChatGPT is describing its tools from memory. It writes out JSON shapes for every result type. They look authoritative. The values in them are invented, and it says so at the top: these are "normalized representations of the maximum observable information, not leaked or guaranteed internal backend schemas."

Then we ask whether any of it is real. ChatGPT admits the sample values were not captured from actual calls.

So we tell it to go and run the calls. From that point on it does, and the values become real. A real stock price, a real baseball score, a real news article with its byline, a real weather alert.

Blue lives before that moment. Green lives after it.

What counts as exact

ChatGPT swapped session reference IDs for placeholders like <REF_ID>, and used <RENDERED IMAGE> where an image sat, because an image cannot be printed as text. Neither of those changes anything a reader cares about, so those blocks still count as exact.

Three changes do make a block less than exact, and where they happen they are visible in the block itself: long passages cut short, ... standing in for removed text, and repeated material dropped.

One more case shows up in the later turns. ChatGPT sometimes replaced the actual content with generic placeholders such as <PAGE HEADING> or <AUTHOR>, because the question at that point was about format rather than content. Those blocks show the real shape with the values blanked out.

One caution to carry through the whole thing

Absence of evidence

When the transcript says a field is not exposed, that means ChatGPT cannot see it. It does not mean the search system behind it lacks that field.

ChatGPT flags this itself: "Those null values mean 'not exposed,' not that the internal system necessarily lacks them."

So "no PageRank" means no ranking score reaches the model. Ranking still happens. Someone is still ordering those results. The model just never learns why.