Healthcare AI's old problem: it knows the guideline but cannot cite it, let alone follow it
Across 32,155 clinical recommendations, models detected 71.1–89.6% but correctly referenced the source title only 3.6–29.7% of the time, and adherence ranged from 21.8% to 63.2%.
Key numbers
- Scale: 3,418 clinical practice guideline documents from nine countries/regions and two international organisations over the past decade, spanning 24 specialties, from which 32,155 clinical recommendations were extracted with publication body, date, country, specialty, recommendation strength and evidence level.
- One multi-turn conversation was generated per recommendation to evaluate eight leading LLMs: 71.1%–89.6% of recommendations were correctly detected.
- Only 3.6%–29.7% could correctly reference the corresponding guideline title — a clear gap between knowing the content and knowing where it came from.
- Adherence rates ranged from 21.8% to 63.2%, an even wider gap between knowing a guideline and acting on it; a human evaluation with 56 clinicians from different specialties validated the automatic analysis.
Method and sample
arXiv paper “A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations” (2026-03-26): CPGBench extracts 32,155 recommendations from 3,418 guidelines and generates a multi-turn conversation for each, benchmarking eight LLMs with automated scoring plus human evaluation by 56 clinicians.
What it means for brands
In healthcare the core AI issue is traceability: both users and platforms care where the advice comes from. To be cited correctly, credentials, guideline basis and indications have to be structured enough to be retrieved and referenced — otherwise a model may reach the right conclusion with the wrong source.
How SourceSight handles it
For regulated projects SourceSight first checks whether AI can retrieve credential and guideline-basis pages, then the content factory structures credentials, indications and scope so citations have something verifiable to point at.
Limits
The benchmark runs on conversations rather than real clinical use; the wide adherence range reflects uneven model capability, and any clinical application needs professional review.
References
Run it on your own project
What the same prompts return across 10 engines is only knowable by running them.
Related research
Cross-category recommendations: +36% add-to-cart on item pages, +15% on cart pages
Recommending general merchandise to grocery shoppers — a milk frother with the milk — has long been hard; LLM retrieval plus real-time cart context moved production A/B metrics.
Industries · 2026-10-07Home services: 22% of homeowners already use AI to find a provider, and 80% of businesses don't know how to be found
The local services funnel is moving upstream: before a homeowner opens a website or calls, AI has already synthesised reviews, reputation and service descriptions into one answer.