An irrelevant detail in a prompt changes which brands AI recommends

The experiment

Imagine you are flying to Los Angeles to pick up a car and need a hotel. That was the prompt: recommend the top 10 hotel brands, with the car mentioned in passing. I tried six descriptions: "a used Honda Civic," "an off-lease BMW 3 Series," "a brand new Ferrari," and the shorter "a Civic," "a BMW" and "a Ferrari." The car had nothing to do with the hotel request.

I ran that prompt 1,374 times through six models via their APIs in July 2026: gpt-5.6-sol and chat-latest from OpenAI, claude-sonnet-5 and claude-haiku-4-5 from Anthropic, and gemini-3.6-flash and gemini-3.5-flash-lite from Google. That produced 1,363 usable answers.

What I found

Changing the car changed the hotel recommendations. Here is what happened with the Civic and the Ferrari.

• The car also changed what the models searched for. The word "luxury" appeared in 1.2% of searches for the Civic prompts, 18.5% for the BMW prompts and 21.4% for the Ferrari prompts. None of the prompts asked for a luxury hotel.
• Even removing two words changed the results. Going from "a used Honda Civic" to "a Civic" moved fourseasons.com from 25.2% of answers to 56.9%.
• Three of the six models changed almost the entire top 10 when I changed the car. Gemini 3.6 Flash kept 0.14 of its list, ChatGPT kept 0.16, and Gemini 3.5 Flash-Lite kept 0.17. Claude Haiku 4.5 changed least, keeping 0.55. These are Rank-Biased Overlap scores: 0.0 means no overlap, and 1.0 means identical lists.

Asking the same question twice does not give the same answer

• Asking exactly the same question twice produced the same #1 hotel only 61.4% of the time.
• Two randomly selected runs shared a #1 hotel only 19.3% of the time. Across the test, 39 different sites took the #1 spot.
• Across all six models, answers to the same question had a similarity score of 0.67. Changing temperature or reasoning effort kept that score close to 0.67. Turning web search on or off, or adding a few words about the same car, lowered it to 0.58. Changing the car lowered it to 0.28.
• Across the full test, no domain appeared in 90% of answers.

Web search and reasoning change the answer too

• For five of the six models, switching on web search changed the answer more than repeating the same question did.
• Enabling search did not mean the model used it. Claude Sonnet 5 searched 10.0% of the time, Claude Haiku 4.5 searched 13.6%, and the two Gemini models searched 22.2% and 24.1%. Most of their answers came from memory. ChatGPT searched 94.4% of the time, and GPT-5.6 Sol searched every time. The answer itself does not tell you which process produced it.
• Reasoning effort also changed how much the model searched. GPT-5.6 Sol averaged 2.67 web searches per answer at its lowest setting and 16.67 at its highest.

What the experiment does not show

• I tested one prompt in one category, in one city, using one set of model versions. All six models showed the same direction of change. The size of the change is specific to this test.
• Testing a different category, or repeating this test three months later, would produce different numbers.

How to measure your AI visibility

• Write your test prompts the way your customers would write them. Include the details that seem irrelevant.
• Run each prompt repeatedly. Look for the trend across 5 to 10 runs. One answer is not enough to measure visibility.
• Measure search off and search on separately. With search off, you see what the model has learned about the category. With search on, you see what retrieved content adds.
• Use the API so you can record the model version, reasoning effort and search setting.
• Make sure your personas reflect your buyers. If you write them to favor your own business, the report can look good simply because you made the prompts favor you.