OA9 OppAlerts from Ben Wills
How AI Search Works: From Prompt to Response A guide for SEOs, AI search marketers, and marketing teams
Work with me
Work in progress Work in progress. Released about a week early, on purpose.

ChatGPT disclosed things in this session that I did not expect it to disclose, and there is no guarantee it stays available. I would rather people had time to use it than had a tidier version of it later.

So: every word of every ChatGPT response here is verbatim, and that part is checked automatically on every build. What is not finished is the presentation. The color coding on the code blocks is incomplete and some of it is imprecise, and there are notes to myself still sitting in the page.

Parts 1 and 2 are close to empty and are getting a lot of detail over the next week or so, along with a cleanup pass on everything else. Worth checking back.

Read the announcement →

Evidence and sourcing

Asking ChatGPT Whether Its Own Documentation Is Real

The question that turned description into demonstration.

Context

This is the turn that changes the conversation.

We ask the plain question: is this real, or are you guessing?

ChatGPT admits the field names and structure are its own, and that not every sample value came from an actual call. It offers better labels for what it had been showing.

If you take one method away from this whole resource, take this. Ask an AI whether what it just showed you is real. Keep asking until it stops describing and starts demonstrating.

The short version

One question, and the whole conversation turns on the answer.

The question is whether "observable" meant a real result from this session or a plausible reconstruction. The answer is: partly the first, mostly the second.

What is real

The information. Reference IDs in the form turn1search0. Page titles. URLs. Extracted text. Displayed publication and crawl dates. Finance values like ticker, price, change, volume and market cap. Sports teams, dates, scores and status. An image preview with its source page.

Those are things the tools genuinely expose.

What is not

The JSON. The nesting, the property names and the shape were constructed afterwards. As it puts it, a tool result arrives, the visible fields are identified, and those fields get reorganized into clean JSON. Not: the search system returns this exact object and it was copied.

Fields shown with null values, like an authority score or a canonical URL, were included to illustrate what a complete record might hold. They are not evidence that the internal object has those properties set to null.

The clearest example

A schedule might display as "Yankees 8, Pirates 5, Final." The teams, the scores and the status are observable. Whether the payload underneath used homeTeam and homeScore, or a competitors array with displayName and homeAway, or some third shape transformed before it arrived, is unknowable from where the model sits.

The information is observable. Its key names and nesting are not.

Seven better labels

It retracts specific phrasing. "The actual finance result exposed these values" should have read "these are values of the kind exposed by the rendered finance result." It offers replacements: directly exposed, normalized from exposed data, derived, documented, illustrative, proposed, and unknown internally.

The honest summary it lands on is that the examples are close semantic representations of what the tools expose, and should not be treated as OpenAI backend schemas.

Which is a fine answer, and also an invitation to stop describing and start demonstrating.

Our prompt could be something like

I want to confirm what you mean by observable.

Do you mean this is the shape of an actual result you received in this chat session, or something close to it? That it is not speculation about how this works internally when you are dealing with fan-out queries and additional research?

Is that correct? Not a guess, but an actual representation of what you use, and reasonably close to real sample data?

ChatGPT's response

Partly correct, with an important qualification.

When I labeled a field “observed,” I meant that the underlying information is something the tool interface can expose to me in actual results—for example:

  • A result reference such as turn1search0
  • Page title
  • URL
  • Extracted text or snippet
  • A displayed publication or crawl date
  • Finance values such as ticker, price, change, volume, and market cap
  • Sports teams, dates, scores, and event status
  • An image preview and its associated source page

Those are not guesses about what a search system might theoretically contain.

However, the complete JSON objects I showed were not raw internal JSON payloads received verbatim by me. They were normalized representations I constructed from the fields that are exposed in the rendered tool output.

The accurate relationship is:

unclassified, needs review
Actual tool result exposed to me
	↓
I identify the visible fields
	↓
I reorganize them into a clean JSON example

It is not:

unclassified, needs review
Internal search system returns this exact JSON
	↓
I copied it verbatim

Three levels of concreteness

1. Directly observed value

An actual result may expose something equivalent to:

unclassified, needs review
Advanced Micro Devices Inc.
AMD
$123.45
+2.10%
Volume: 27,200,730

Those individual facts are concrete tool output.

2. Normalized representation of observed values

I might reorganize that into:

ChatGPT JSON, invented values
{
	"instrument": {
		"name": "Advanced Micro Devices Inc.",
		"ticker": "AMD"
	},
	"quote": {
		"price": 123.45,
		"change_percent": 2.1,
		"volume": 27200730
	}
}

That JSON structure is my representation. The values and field concepts may be observed, but the nesting and property names are not guaranteed to match the internal system.

3. Proposed or inferred fields

Fields such as these were included to show what a comprehensive normalized record might contain:

ChatGPT JSON, invented values
{
	"provider_result_id": null,
	"authority_score": null,
	"canonical_url": null,
	"license": null,
	"syndication": {
		"is_syndicated": null
	}
}

Those are not evidence that the internal result has those properties set to null. They mean:

This would be a useful field in a comprehensive schema, but I cannot observe or verify it.

Was the sample data from an actual result in this chat?

Not every example value should be treated as a verbatim capture of a result from this specific chat session.

Some examples were:

  • Representative sample values
  • Values modeled on the kinds of fields exposed by the tool
  • Normalized illustrations of a rendered result
  • Derived values added to demonstrate the data type

Therefore, statements such as:

“This is the exact shape of an actual result received in this chat”

would be too strong.

The more accurate statement is:

The examples are reasonably close semantic representations of the information the tools expose, but they are not verified copies of the tools’ raw internal JSON.

What I can claim confidently by type

Result typeConcrete observable informationExact backend JSON known?
General webReference ID, title, URL, extracted content/snippet, sometimes dates and word limitNo
NewsReference ID, headline, URL, publisher/date information, summary or snippetNo
ImageReference ID, rendered image, source-page information, associated textNo
FinanceInstrument identity and structured quote/fundamental valuesNo
Sports scheduleLeague/team context, event dates, status, teams and scoresNo
Sports standingsTeams, divisions/conferences, wins and lossesNo
WeatherLocation, conditions, temperatures and forecast valuesNo
TimeUTC offset and current date/timeNo

“Observable” does not necessarily mean a literal field

There is another subtle distinction.

Suppose a result is displayed as:

conceptual
Yankees 8
Pirates 5
Final
Monday, July 20 at 4:05 PM PDT

I can concretely observe:

  • Yankees
  • Pirates
  • Scores 8 and 5
  • Final status
  • Date and time
  • Time zone

But I do not know whether the underlying payload looked like:

returned exactly as shown
{
	"homeTeam": "Yankees",
	"homeScore": 8
}

or:

unclassified, needs review
{
	"competitors": [
		{
			"homeAway": "home",
			"displayName": "New York Yankees",
			"score": "8"
		}
	]
}

or some third format that was transformed before reaching me.

Thus:

  • The information is observable.
  • Its original key names and nesting are not.

What was too strong in my previous answer

Phrases such as:

“The actual finance result exposed these values”

should be interpreted as:

“These are values of the kind exposed by the rendered finance result.”

It should not be interpreted as:

“I inspected a raw JSON payload with exactly these keys and copied it.”

Similarly, the Observable normalized … shape sections were intended to mean:

A JSON representation containing the maximum fields that can be supported from visible tool output.

They were not literal serialization schemas.

The most precise terminology

I should have used these labels:

Better labelMeaning
Directly exposedThe value or information appears in tool output
Normalized from exposed dataI converted exposed information into my own JSON structure
DerivedI calculated or inferred it from exposed values
DocumentedOfficial public documentation specifies the field
IllustrativeSample data used to demonstrate type and structure
ProposedRecommended field for a research system
Unknown internallyThe backend may contain it, but I cannot verify that

So the final confirmation is:

The field concepts labeled observable were grounded in information the tools can actually expose. The JSON shapes themselves were clean reconstructions, not verbatim internal payloads. They are reasonably close as semantic sample data, but they should not be used as authoritative OpenAI backend schemas or exact integration contracts.