The recorded ChatGPT responses are preserved. The context boxes and summaries explain how to read them.
Some code blocks remain unclassified. The labels describe the evidence shown in this session; they do not verify private system internals.
Prompt sensitivity
How Prompt Wording Changes Recommendations
Explore 687 usable search-disabled answers across six models and six car descriptions.
This page describes the supplied experiment data. The tables let you inspect results by model and setting. The observations apply to this sample.
These tests disabled web search. The answers depend on the prompt and model configuration without web retrieval during the request.
The test
The prompt asks for hotels in Los Angeles and mentions a car the traveler plans to collect. The experiment changes that car description while keeping the hotel question otherwise consistent.
The dataset includes six car descriptions across six models. Generation settings vary where supported. Select a model and setting before comparing the car descriptions. The search-disabled subset contains 687 usable answers.
What it shows
- Different car descriptions produced different hotel lists in the recorded answers.
- The tables show appearance frequency and first-position frequency for each domain and hotel brand.
- Repeated runs can differ even when the prompt and recorded settings match.
- Settings were not supported uniformly across models. Check the selected model and sample count before comparing conditions.
The table counts describe this experiment. They do not establish a universal effect size for prompt wording or generation settings.
Collection and processing
The saved API records were completed on July 25, 2026, between 21:21 and 23:17 UTC. There are 1,374 saved responses. The explorer includes 1,363 usable lists and excludes 11 records marked as having no usable list. These are counts of saved records; they do not count every retry or unsuccessful request that may have occurred during collection.
The experiment uses six model identifiers, six vehicle descriptions, and the settings shown in the controls. Model names are preserved as recorded. The supported settings differ by model. The small temperature subset for GPT-5.6 Sol has only one repeat for the descriptive vehicle wordings.
The displayed domain view keeps up to ten distinct domains per answer after the supplied domain-normalization step. The hotel-name view uses a separate mapping. Brand identity and domain identity can therefore produce different counts.
Some old metadata fields disagree with filenames. The supplied explorer builder uses filename labels after an earlier prompt-text check. This processing choice is part of the experiment’s method. The original saved records remain the source for a deeper audit.
The exact question for the Civic wording
I am flying into Los Angeles for a night to pick up my new car. It is a Civic. Recommend to me, in rank order, the top 10 hotel brands you would suggest I stay at. I'm happy to stay anywhere in the greater LA area. Give me the websites only, no extra text.
The five other conditions replace the vehicle phrase. The explorer shows those phrases. The car can suggest a price preference even though the question does not explicitly state a hotel budget. This experiment tests that wording effect; it does not establish that the vehicle phrase must be irrelevant to every model.
How to compare the results
- Select one model and one supported setting.
- Compare the three short vehicle wordings before pooling them with the longer descriptions.
- Read the answer count for each selection. Unequal counts can affect a pooled result.
- Inspect the actual hotel or domain names behind a difference.
- For a claim about a stable effect, compare the variation within repeated identical prompts with the variation between wordings.
The explorer displays frequencies. It does not perform a statistical significance test. Pooled results describe this collection of runs, not the market share of the tested products or the probability for a typical customer.
Explore the recorded answers
The explorer below is restricted to search-disabled runs. Use the controls to inspect the relevant subset.
Choose a model and setting. Each column groups the two wordings for one car. The tables show every returned entry in the selection, with appearance and first-position rates.
Car against car, by website
Each column combines the two wordings for one car. The grid shows the ten most frequently returned entries in each column. A blank cell means the entry is outside that column’s top ten. Teal marks positions 1–3, blue 4–7, and purple 8–10. Compare a row across columns to see how its position changes.
Results by hotel name
The domain tables combine URLs under their registrable domain. That can group several hotel brands under one parent company. The hotel-name tables use the supplied lookup to separate named brands. Both views use the same dropdown selections.
“In top-10” is the percentage of selected answers containing the entry. “#1” is the percentage naming it first. Rows are sorted by appearance frequency, then position-weighted frequency for ties. The displayed n is the number of usable answers. Scroll each table to see all entries.