By the numbers
Leading AI legal-research tools built on retrieval-augmented generation still hallucinated 17% to 34% of the time, while general-purpose GPT-4 erred on 58-82% of legal queries (Stanford RegLab/HAI, 2024).
Source: Stanford HAI
On Vectara's HHEM leaderboard, even the top-ranked model still hallucinated in 3.3% of summaries of a source document it was given, and several frontier models exceeded 10%, showing that grounding alone does not guarantee faithful output (Vectara, 2026).
Source: Vectara
Gartner predicted at least 30% of generative-AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, weak risk controls, rising costs, and unclear business value (Gartner, 2024).
Source: Gartner
An MIT NANDA study of 300 enterprise AI deployments found about 95% of generative-AI pilots delivered no measurable P&L impact, while only about 5% created significant value (MIT NANDA, 'The GenAI Divide', 2025).
Source: MIT NANDA
Strong LLM 'judge' models agreed with human preferences over 80% of the time, matching human-to-human agreement, but showed position, verbosity, and self-preference biases, so judges must be validated against human labels (Zheng et al., 2023).
Source: arXiv (Zheng et al.)
The U.S. NIST AI Risk Management Framework structures AI risk into four functions (Govern, Map, Measure, Manage), and its 2024 Generative AI Profile (NIST-AI-600-1) gives enterprises a standard for measuring and governing model risk (NIST, 2024).
Source: NIST
Key terms
- Hallucination (confabulation)
- An output that is fluent and confident but factually wrong or unsupported by any source. NIST's Generative AI Profile calls this 'confabulation' and lists it as a primary generative-AI risk.
- Golden set (labelled evaluation set)
- A curated set of representative inputs paired with known-correct answers. Every model or prompt change is scored against it, turning accuracy into a repeatable number instead of a gut feeling.
- LLM-as-a-judge
- Using a strong LLM to grade another model's outputs against a rubric or reference answer. It scales evaluation cheaply and can approximate human judgment, but carries known biases and must be validated against human labels.
- RAG grounding and faithfulness
- Retrieval-augmented generation feeds the model real source documents so answers can cite evidence. Faithfulness measures whether the answer actually follows those sources; grounding reduces hallucination but does not remove it.
- Regression testing (evaluation harness)
- An automated suite that re-scores the whole system against the labelled set on every change, so a gain in one area cannot silently break another. It is standard software discipline applied to model quality.
- Human-in-the-loop and confidence thresholding
- Routing low-confidence or high-stakes outputs to a person instead of auto-answering. Well-designed systems flag uncertainty rather than guessing, trading a little coverage for far fewer costly errors.
At a glance
| Method | What it measures | When to use | Main limitation |
|---|---|---|---|
| Golden / labelled set | Task accuracy versus known-correct answers | Any production system; the baseline metric | Costly to label and must reflect real inputs |
| LLM-as-a-judge | Output quality versus a rubric or reference, at scale | Grading open-ended answers or high volumes cheaply | Position, verbosity, and self-preference bias; validate versus humans |
| Human-in-the-loop review | Human-judged correctness on sampled or flagged cases | High-stakes or low-confidence outputs | Slow and expensive; cannot cover every case |
| RAG faithfulness check | Whether answers are supported by retrieved sources | Document Q&A, search, and knowledge assistants | Retrieval misses and sycophancy still cause errors |
| Public benchmark / leaderboard | Relative model skill on standardized tasks | Early model shortlisting and sanity checks | Not your data; contamination and overfitting inflate scores |
| Regression test suite | Score changes across every model or prompt update | Ongoing quality control after launch | Only as good as the labelled set behind it |
What to check before you buy
- Ask for the system's measured accuracy on a labelled test set that reflects your real inputs, not demo examples or a public benchmark score.
- Require a clear definition of how 'correct' was scored, who labelled the data, and how large the evaluation set is.
- Confirm hallucination and faithfulness are measured explicitly (are answers grounded in and supported by sources), not assumed away by 'we use RAG'.
- Insist on a regression test suite so every future model or prompt change is re-scored against the labelled set before it ships.
- Check that low-confidence or high-stakes cases are flagged for human review rather than answered with a confident guess.
- Ask how accuracy will be re-measured on live production traffic and how often, since real-world accuracy drifts from test-set accuracy.
- Treat any 'hallucination-free' or guaranteed-accuracy claim as a red flag; even legal RAG tools hallucinate 17-34% of the time (Stanford). Ask for the number and the method.
- Map evaluation to a recognized framework such as NIST AI RMF's Measure function so accuracy, risk, and monitoring are governed rather than ad hoc.
Frequently asked questions
Why isn't a great demo enough to trust an AI system?
Demos use hand-picked inputs; production sends messy, adversarial, and edge-case inputs the demo never showed. Gartner predicted at least 30% of generative-AI projects would be abandoned after proof of concept by the end of 2025, and an MIT NANDA study found about 95% of enterprise pilots delivered no measurable P&L impact. A measured accuracy number on realistic data is the only reliable signal.
What is an acceptable hallucination rate?
There is no universal number; it depends on the use case and the cost of a single error. Even top models still hallucinate in a few percent of summaries of a document they were given (Vectara HHEM), and specialized legal RAG tools hallucinate 17-34% of the time (Stanford). The right target is set per task, measured, and monitored, never assumed to be zero.
Does retrieval-augmented generation (RAG) eliminate hallucinations?
No. RAG grounds answers in real sources and substantially reduces hallucination, but Stanford found even leading legal RAG tools still hallucinate 17-34% of the time, from retrieval misses, out-of-date or out-of-jurisdiction sources, and sycophancy. RAG shrinks the problem; measurement and human review manage what remains.
Can we trust an AI to grade another AI (LLM-as-a-judge)?
As a scalable approximation, yes: strong LLM judges reach over 80% agreement with human preferences, matching human-to-human agreement (Zheng et al., 2023). But they carry position, verbosity, and self-preference biases, so a serious evaluation validates the judge against human labels and does not rely on it alone for high-stakes calls.
Aren't public benchmarks and leaderboards enough?
They help shortlist models but not trust your system. Benchmarks test standardized tasks on someone else's data, and scores can be inflated by training-data contamination or overfitting to the test. What matters is accuracy on your inputs, scored with your definition of correct.
How do I make a builder prove accuracy?
Ask for a labelled evaluation set drawn from your data, the measured accuracy against it, the exact scoring method, and a regression suite that re-scores every change. Honest builders report a measured number and its limits rather than guaranteeing one; CONE RED, for example, ships each system with an evaluation harness and flags low-confidence cases for human review instead of guessing.
What does 'measured accuracy' mean, and why won't a good builder guarantee a number?
Measured accuracy is the system's score on a labelled test set that mirrors your real inputs, using an explicit definition of correct. A responsible builder reports that number honestly, states its limits, and re-measures it as the model and data change, because accuracy is specific to a use case and drifts over time. A promise of fixed accuracy or zero hallucinations is a warning sign, not a feature.
This guide is educational and even-handed: CONE RED is one option among several, and no honest firm guarantees a fixed accuracy, compliance, or business outcome. Every statistic above links to a source you can check; figures are current as of publication and may change.