How to evaluate a conversational AI system: 23 metrics plus agreement with human raters
Single-turn, single-task accuracy cannot capture real use; multi-turn dialogue needs five dimensions, 23 metrics — and proof that machine scoring matches human scoring.
Key numbers
- Framework: derived using the Goal-Question-Metric method across five categories — task performance, factual accuracy, interaction quality, control capability and system efficiency.
- Scale: 23 metrics spanning multi-turn conversations, with an LLM-as-judge pipeline for scalable automated scoring.
- Reliability: validated against three trained human coders; after iterative rubric calibration, twelve of fifteen LLM-scored metrics reached strong agreement (ICC ≥ 0.73), three of them perfect, while the remaining three had near-zero variance in human scores and were reported via mean absolute error (0.04–0.28).
- Effectiveness: 970 dialogues (16 scenarios × five personas) generated and evaluated across four conversational configurations, from a vanilla LLM with raw energy data to a multi-agent system, distinguishing their respective strengths and weaknesses.
Method and sample
arXiv paper “A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems” (2026-09-04): a Goal-Question-Metric framework with 23 metrics in five categories, scored automatically via LLM-as-judge, cross-validated against three trained human coders, then applied to 970 dialogues across four system configurations.
What it means for brands
Evaluating conversational AI properly means multi-turn, multi-dimension, with a human-agreement check: accuracy alone hides problems in interaction quality, controllability and efficiency. Any team building conversational products can reuse this shape.
How SourceSight handles it
SourceSight follows the same discipline: multi-turn runs, per-engine and per-keyword dimensions (mentions, position, sources, coverage), and explicit labelling of which numbers are automated versus human-checked, so one blended score never replaces diagnosis.
Limits
The framework was built for home energy, so metrics and weights may not transfer to other domains; the LLM-as-judge agreement required human calibration, and uncalibrated scoring is unreliable.
References
Run it on your own project
What the same prompts return across 10 engines is only knowable by running them.
Related research
Same engine, different surface, different answer: test app, web and API separately
On the same platform and the same questions, app and web cite systematically different source sets — and the API shows something else again.
Methods · 2026-10-07How to design a retest you can trust: five moves so noise does not pass as progress
AI visibility is a distribution, not a point: fix the prompts, repeat the runs, add controls, paraphrase, validate by hand — or you cannot attribute the change.