Why one measurement is not a measurement: AI visibility is a distribution
Ask the same question twice and the brands and rankings change — a single observation cannot tell you where you stand.
Key numbers
- A study on measuring GEO concludes answers vary across runs, prompts and time, making one-off observations unreliable: brand visibility should be characterised as a distribution, not a single point.
- A parallel evaluation study shows the same mechanism: repeated responses to the same query produce different brands and rankings, so evaluation requires repeated sampling rather than a single generated list.
- That study sampled six LLMs across five product categories and found category-level queries systematically omit some established brands — recommendation prominence tracks search interest and online brand conversation more than conventional brand popularity.
- The same work shows the retrieved brand set changes when a user's constraints (budget, use case, limits) are included, and that brands missing from ordinary recommendations can become retrievable when distinctive cues are supplied.
Method and sample
Two arXiv papers: one on the statistical nature of GEO measurement (19 pages, several repeated-measurement designs) arguing visibility is a distribution; the other proposing repeated-sampling evaluation of open-ended brand recommendations with BRP@k and MRR@k across six LLMs and five categories.
What it means for brands
“I asked once and AI didn't mention me” is not a finding, and neither is a single citation. Visibility is a distribution: repeat the runs, look at the range, then look at the trend. The same applies to improvements — a single better number can be noise.
How SourceSight handles it
SourceSight prompt libraries are grouped by how customers actually ask, and every audit runs the full set and records mention rate and position. Snapshot comparison shows movement across the same prompts rather than one-off results.
Limits
Both papers work on experimental or panel data that skews to English and Western markets; how wide the distribution is depends on engine and industry and cannot be transferred directly to any single project.
References
Run it on your own project
What the same prompts return across 10 engines is only knowable by running them.
Related research
Same engine, different surface, different answer: test app, web and API separately
On the same platform and the same questions, app and web cite systematically different source sets — and the API shows something else again.
Methods · 2026-10-07How to design a retest you can trust: five moves so noise does not pass as progress
AI visibility is a distribution, not a point: fix the prompts, repeat the runs, add controls, paraphrase, validate by hand — or you cannot attribute the change.