OA9 OppAlerts from Ben Wills
How AI Search Works: From Prompt to Response A guide for SEOs, AI search marketers, and marketing teams
Work with me
Work in progress Work in progress. Released about a week early, on purpose.

ChatGPT disclosed things in this session that I did not expect it to disclose, and there is no guarantee it stays available. I would rather people had time to use it than had a tidier version of it later.

So: every word of every ChatGPT response here is verbatim, and that part is checked automatically on every build. What is not finished is the presentation. The color coding on the code blocks is incomplete and some of it is imprecise, and there are notes to myself still sitting in the page.

Parts 1 and 2 are close to empty and are getting a lot of detail over the next week or so, along with a cleanup pass on everything else. Worth checking back.

Read the announcement →

Evidence and sourcing

ChatGPT Confirms What Was Real in the Session

The evidence ladder, restated in one place.

Context

The last check. We ask whether everything in the conversation was an actual internal representation, and whether the exceptions were flagged.

The answer is a short summary of the same distinction ChatGPT had been drawing since the beginning, restated in one place.

The short version

A direct question and a direct answer. No, not everything described was an exact internal representation, and it should not be characterized that way.

The breakdown

The cleanest statement of the whole conversation, in one place:

or shown with placeholders.

not internal payloads.

exposed.

It also admits it was not perfectly clear throughout. Earlier phrases like "observable normalized shape" and "the actual result exposed these values" could reasonably be read as claiming access to raw internal JSON, and that was too strong.

The rule to carry

Unless a model shows a raw textual tool response and labels it as a close transcription, any JSON it produces should be treated as its normalization, illustration, derivation, or a published schema. Not a verbatim internal representation.

That is not a retraction of the evidence. The requests are still real, the returned text is still real, and the formats are still reproducible by anyone who asks. It is a boundary around what the evidence supports, drawn by the thing producing it, which is a better outcome than a confident answer would have been.

Our prompt could be something like

I want to confirm that everything you are saying is actually your internal representation and not a guess. And that where you have communicated something that is not exact, you have been clear about it throughout this chat. Is that correct?

ChatGPT's response

No. Not everything I described was an exact internal representation, and I should not characterize it that way.

The accurate breakdown is:

  • Exact request JSON: The tool arguments shown were the actual structured requests sent.
  • Model-visible response examples: These were close transcriptions of what the tools returned to me, but some were shortened, anonymized, or represented with placeholders such as <REF_ID> and <RENDERED IMAGE>.
  • Normalized JSON: These objects were created by me to organize the returned text. They were not internal payloads I received.
  • Documented schemas: These came from public OpenAI documentation, not from inspecting private internal objects.
  • Derived fields: These were calculated or inferred from visible values.
  • Proposed or unknown fields: These illustrated a useful research schema or identified data that was not exposed.

I have tried to mark those distinctions, particularly in the later responses, but I was not perfectly clear throughout the entire conversation. Earlier wording such as “observable normalized shape,” “exact model-visible format,” and “the actual result exposed these values” could reasonably be interpreted as claiming access to raw internal JSON. That was too strong.

The reliable rule is:

Unless I explicitly show a raw textual tool response and label it as a close transcription, any JSON object I produced in this conversation should be treated as my normalization, illustration, derivation, or a publicly documented schema—not as a verbatim private internal representation.