The public conversation about AI is dominated by questions of intelligence. Is the latest model smarter than the last one? Does it reason? Is it approaching human-level ability? Leaderboards rank models on exams and puzzles, and every launch comes with a chart showing a higher score.
Updated September 2026: we added sources on benchmark contamination, model calibration and why training can reward guessing.
These are interesting questions. For almost everyone actually using AI, they are the wrong ones.
Capability is not reliability
A model can be capable of solving a hard problem and still fail at an easy one. Anyone who uses AI tools daily has seen this: an assistant writes elegant analysis, then miscounts items in a list or misreads a date. The same prompt can produce a great answer on Monday and a flawed one on Tuesday.
For real work, the question is not “can it do this?” but “how often does it do this correctly, and what happens when it does not?”
Why benchmarks mislead
- They measure peaks, not consistency. A high score does not tell you how often a model fails on your kind of task.
- They can be saturated or contaminated. When models are trained on data that overlaps with test questions, scores can overstate real ability, a problem researchers call benchmark data contamination.
- They do not match your work. Your documents, customers and edge cases are not in any benchmark.
Benchmarks are useful for researchers tracking progress. They are poor guides to buying decisions.
What reliability actually requires
- Consistency on the tasks you care about
- Calibrated confidence: saying when it is unsure rather than guessing; Anthropic research found that large models are often well calibrated about whether they know an answer, which suggests this is achievable
- Verifiability: sources, reasoning or tests that let you check the answer quickly
- Graceful failure: when it cannot do something, it says so
Notice that none of these require the model to be smarter. Several require it to be more honest.
A practical test Before relying on an AI tool for a recurring task, run it on twenty real examples and check every result. Your error rate on your own work is worth more than any published score.
What I want from AI companies
Publish error rates for common tasks, not just benchmark wins. Build interfaces that show sources and uncertainty. Reward models for saying "I don’t know". OpenAI researchers made this case in 2025, arguing that evaluations which penalize uncertainty encourage models to guess. Our guide to writing good prompts already recommends asking models to flag uncertainty; the products should make that the default.
Why this matters now
As AI moves from answering questions to taking actions, as agents do, reliability stops being a nice-to-have. An unreliable answer wastes a few minutes. An unreliable action can send the wrong email, delete the wrong file or pay the wrong invoice.
The next great leap in AI will not be the one that tops a leaderboard. It will be the one that makes these tools boringly, verifiably dependable, the kind of progress we praised in our column on boring AI.
Sources
- Benchmark Data Contamination of Large Language Models: A Survey, arXiv, 2024
- Language Models (Mostly) Know What They Know, arXiv, 2022
- Why language models hallucinate, OpenAI, September 2025
