Same engine, different surface, different answer: test app, web and API separately
On the same platform and the same questions, app and web cite systematically different source sets — and the API shows something else again.
Key numbers
- Chinese generative search: four mainstream platforms across web and app — eight interfaces, 614 queries, three replications per combination, cleaned from 214,119 raw records into 160,860 citation-level records; one conclusion is that app and web source sets differ systematically for the same platform.
- Commercial recommendation audit: for the same query, the ChatGPT and Gemini product interfaces share only 5.4% of cited domains on average, with no common domain in 76.7% of comparisons.
- Same model, different surface: API versus interface domain overlap is just 12.0% for ChatGPT and 14.8% for Gemini, and the types and layers of source information differ too.
- The surface also changes trust: the same model delivering health information as text, speech or an embodied interface produced significant differences in user trust (N=20).
Method and sample
Synthesis of three studies: the large-scale audit of Chinese generative search (arXiv:2607.15771, eight interfaces × 614 queries × three replications), the AI product recommendation audit (arXiv:2609.18729, interface versus API) and the health-information trust study (arXiv:2608.21177, three dissemination interfaces).
What it means for brands
“How do we perform in AI” is meaningless without naming the surface: testing only web misses the app, testing only the API misses what users actually see. Sources and recommendations differ across those surfaces, so blending them into one number hides the problem.
How SourceSight handles it
SourceSight records mentions, position and cited sources per engine and states which surface a run used; cross-engine comparisons carry the same caveat — different engines and different surfaces should not be merged into a single score.
Limits
The three studies differ in subject, industry and language and do not corroborate one another directly; the cause of app-versus-web differences is unpublished, so this is grounds for separate testing rather than an explanation.
References
Run it on your own project
What the same prompts return across 10 engines is only knowable by running them.
Related research
How to design a retest you can trust: five moves so noise does not pass as progress
AI visibility is a distribution, not a point: fix the prompts, repeat the runs, add controls, paraphrase, validate by hand — or you cannot attribute the change.
Methods · 2026-10-07What GEO methods actually hold up: 45 studies and a cold shower
The widely cited “40% visibility lift” comes with a condition: the source was already in a fixed context. It proves neither discoverability nor durable traffic.