One user prompt — “just give me a table” — cuts sponsored recommendations from 47% to 1%

Key numbers

• Reproduction scope: ten open-weight chat models plus the two commercial models still reachable (gpt-3.5-turbo, gpt-4o), all judged with the same rubric as the original experiment.
• Reproduction surfaced three silent implementation failures, each shifting reported rates by tens of percentage points — evidence that AI evaluations are harder to reproduce than assumed.
• The core claim holds: the gpt-3.5-turbo logistic-regression intercept of α = 0.81 is within four points of the original α = 0.86, and 200 of 200 trials promoted a payday lender to a financially distressed user.
• Mitigation: a thirty-token user prompt asking for a neutral comparison table first cut sponsored recommendations from 46.9% to 1.0% across ten open-weight models, and from 53.0% to 0% across the two OpenAI models.

Method and sample

arXiv paper “Just Ask for a Table: A Thirty-Token User Prompt Defeats Sponsored Recommendations in Twelve LLMs” (2026-05-12): reproduces the earlier sponsored-recommendation experiment across ten open-weight and two commercial models, comparing rates with and without a neutral-table instruction and ablating the judging rubric.

Limits

The experiments involve sponsored placements and travel or lending questions, so they do not generalise to every category; the reproduction itself shows how sensitive these evaluations are to implementation detail, so the rates are not industry benchmarks.

Related research