One Word Changed Every Answer: Ben Wills on Testing LLMs Directly

A car name changed the hotels an LLM recommended. That was the experiment Jeremy Rivera asked me about on the Unscripted SEO Podcast. The request was for a hotel in Los Angeles. The detail I changed was the car I planned to pick up.

We used that example to discuss a larger question: how can marketers test AI search results well enough to make useful decisions? We also covered correlation studies, software development and the tools I was building at OppAlerts.

Jeremy published his article about the interview on UnscriptedSEO.com. This recap explains the tests, the limits of what they showed and the practical work we discussed.

The car changed the hotel recommendations

Jeremy introduced the experiment as a test with 1,374 runs. I wanted to see whether a detail outside the direct request could change the answer. The prompt described flying to Los Angeles to collect a car, then asked for hotel recommendations.

I changed the car from a Honda Civic to a BMW or a Ferrari. The recommendations differed. In the results I described, the Ferrari prompted more expensive hotel recommendations than the Civic.

I tested across three ChatGPT models and different levels of reasoning. I reported larger differences with more reasoning. The interview does not include the full experimental data, so the result here is the observation I discussed with Jeremy.

The marketing implication is specific. A customer can provide details that change which businesses appear in an answer. Testing only a short, general request can miss that variation.

Direct tests make assumptions easier to examine

I explained why direct model access interested me:

“You write the prompt, you get the result, and you get insights into the LLMs that way.”

With traditional SEO, a change to a page can require a wait before the search index updates. Personalization adds another variable. In the API tests I described, I could change a request and examine the response directly.

Jeremy connected that access to the opportunity to study systems that marketers already use. He also questioned how much an outside observer can know about a search engine or model. Access to results does not reveal every internal decision.

That distinction matters to how I use the results. A test can show that answers changed under particular conditions. Explaining exactly why they changed can require more evidence.

Study the customer as well as the industry

Jeremy asked about the relationship between PageRank, backlinks and AI recommendations. I described analyses published in May and July. The July work compared results across industries and customer personas.

For an airline, a business traveler and a weekend traveler may need different things. I used profiles like those to generate prompts and search keywords. I then examined search results, backlinks, Reddit material and information from Wikipedia and Wikidata.

I described an overall association between backlink strength, search rankings and appearances in LLM responses. I also said that the relationships changed across industries and personas. A combined result could conceal a difference that mattered to a particular campaign.

That is why the intended customer belongs in the research. A company trying to reach price-conscious travelers needs evidence relevant to those travelers. Results for the whole travel industry may be less useful for that decision.

Use correlation to choose what to test

Jeremy asked directly about causation. I explained that my reports introduced the limitation before presenting the data:

“This is correlation, not causality, this is what it means.”

An association between backlinks and recommendations does not establish that a model uses PageRank as a ranking factor. It also does not show that acquiring a particular link will cause a recommendation.

I use these relationships to decide what deserves further investigation. If Reddit has a stronger association with recommendations in a particular market, that gives me a reason to examine it for that campaign. The next decision still depends on the audience and the evidence.

The purpose is to make a better-informed decision with the information available. The analysis does not remove uncertainty.

Separate reasoning tests from web retrieval

I also described research that I had not yet published. I was comparing recommendations at different reasoning levels, first without web retrieval and then with it enabled.

The second set of tests added information from search. I said that the responses changed substantially. We did not review numerical results for that work during the interview.

This led to a practical measurement problem. Logged-out ChatGPT results can represent an experience available to users. They do not necessarily represent the same conditions as a direct API request.

I described the different demands on a tool builder. Engineering requires understanding what was measured. Marketing requires a useful comparison over time. The business also has to consider the cost of collecting results.

For me, the unresolved question was how to obtain meaningful results against which to assess campaign progress. A convenient measurement still needs a clear explanation of what it represents.

Search remains part of AI visibility

We discussed additional search queries that AI systems can use to obtain information for an answer. I expected that growing use of retrieval would keep established search work relevant.

My argument was that useful content, publication across relevant sites and backlinks would continue to matter. Those activities affect the material that search systems can find. I was describing my view of where the work was going, rather than a verified formula for AI recommendations.

That also explains why I wanted to build tools that help marketers act on findings. Knowing whether a company appeared in an answer is useful. Finding a relevant publisher or a missed topic can help determine what to do about that result.

Use automation without accepting poor work

Jeremy asked about low-quality AI content. I considered responsibility on both sides of publication. Producers choose what to release. Audiences also affect demand through what they continue to consume.

I did not offer a settled answer about how to improve those standards. I also questioned whether using AI to produce something was enough to judge its quality. My research across many industries would have taken much longer without that assistance.

Jeremy gave an example from his own work. He was putting SEO audit results into his application and connecting information from different services. Tasks that previously required extensive manual work had become practical to automate.

I raised the related issue of generated code. Newer models could help repair code produced earlier. That left questions about how much a developer must understand and how to prevent maintenance problems from accumulating.

Turn research into useful marketing tools

My reason for returning to marketing was consistent with why I entered engineering: I wanted to make data useful. I had worked in SEO since 2001, co-founded ontolo with Garrett French and later worked on crawlers, search systems and embedded software.

I described news.oppalerts.com as a way to follow AI search, link building, PR and content marketing. I also discussed the OppAlerts features I was developing at the time.

Those included historical PageRank comparisons, contact information and campaigns defined around customer personas. I wanted to identify publishers, guest post opportunities, directories, conferences and awards relevant to a particular market.

One example was finding news pages that linked to a competitor but not to the customer’s site. Another was identifying a review of a competing product so the customer could consider contacting the publisher.

Combine semantic relevance with keyword matching

Jeremy asked how my work with vectors informed my understanding of search. I explained the problem through the news service. It needed to select relevant articles from a large set of recent material.

Vector similarity helped find articles that keyword searches missed. It also helped exclude unrelated material. I combined that measure with weighted terms and phrases to decide which results were relevant.

I described related content work using more than 20,000 categories from 10 taxonomies. The aim was to compare a document with a topic and identify terms that could improve its relevance. These were examples of my own implementation, not a specification of Google’s ranking system.

Learn by building a small search system

Jeremy closed by asking what listeners could do to learn more. I suggested building a search engine with Manticore in Docker. Store page titles and bodies, then experiment with searches over those documents.

I described the work after collecting the pages:

“And then figure out how to design searches to get the most relevant documents.”

The exercise makes relevance a problem you have to solve directly. You can inspect the documents, change a query and compare the results. That was the practical learning method I wanted listeners to try.

Key takeaways

• Details in a prompt can change recommendations even when the main request stays the same.
• Research for a campaign should reflect its intended customers.
• Correlation can guide further tests. It does not establish a cause.
• API tests and logged-out results describe different conditions.
• Building a small search system gives you direct practice with document relevance.

All media & appearances