ChatGPT disclosed things in this session that I did not expect it to disclose, and there is no guarantee it stays available. I would rather people had time to use it than had a tidier version of it later.
So: every word of every ChatGPT response here is verbatim, and that part is checked automatically on every build. What is not finished is the presentation. The color coding on the code blocks is incomplete and some of it is imprecise, and there are notes to myself still sitting in the page.
Parts 1 and 2 are close to empty and are getting a lot of detail over the next week or so, along with a cleanup pass on everything else. Worth checking back.
Prompt sensitivity
How One Word in a Prompt Changes AI Recommendations
1,363 answers across six models, with search turned off and one irrelevant word varying between runs.
This section is not written yet. The research behind it is finished and the numbers below are verified, but the writing and the exhibits are still to come.
This part covers what happens when there is no search at all. The question goes to the model, the model answers from what it already knows, and nothing is fetched.
The test
One prompt asks for hotel recommendations in Los Angeles. It mentions, in passing, what car the traveler is flying in to pick up. The car has nothing to do with hotels.
Six wordings for the car, from "a used Honda Civic" through "a brand new Ferrari", plus one-word versions so that some pairs of prompts differ by a single word. Six models, three runs each, with temperature and reasoning level swept where the model supports them.
What it shows
- The Ferrari and used-Civic answers share zero domains in their pooled top seven.
- fourseasons.com appears in 25.2% of used-Civic answers and 99.1% of Ferrari answers. bestwestern.com appears in 79.2% of used-Civic answers and 0.0% of Ferrari answers.
- Dropping one word, from "a used Honda Civic" to "a Civic", takes fourseasons.com from 25% to 57%.
- Asking the identical question twice gives answers that agree about 0.67 of the way. Two runs of the same setting keep the same top pick only 62% of the time.
- Reasoning level, which most people never set and cannot see, flips whole answers. One model recommends luxury hotels for the used Civic at its lowest reasoning setting and mainstream chains at every setting above it.
- Temperature, the dial everyone assumes matters, is indistinguishable from ordinary run-to-run noise.
Full write-up to come.