The public conversation about AI is dominated by questions of intelligence. Is the latest model smarter than the last one? Does it reason? Is it approaching human-level ability? Leaderboards rank models on exams and puzzles, and every launch comes with a chart showing a higher score.
These are interesting questions. For almost everyone actually using AI, they are the wrong ones.
Capability is not reliability
A model can be capable of solving a hard problem and still fail at an easy one. Anyone who uses AI tools daily has seen this: an assistant writes elegant analysis, then miscounts items in a list or misreads a date. The same prompt can produce a great answer on Monday and a flawed one on Tuesday.
For real work, the question is not “can it do this?” but “how often does it do this correctly, and what happens when it does not?”
Why benchmarks mislead
Benchmarks measure peaks, not consistency, so a high score doesn’t tell you how often a model fails on your kind of task. They can also be saturated or contaminated. When models are trained on data that overlaps with test questions, scores can overstate real ability, a problem researchers call benchmark data contamination. And they don’t match your work. Your documents, customers and edge cases aren’t in any benchmark.
Benchmarks are useful for researchers tracking progress. They are poor guides to buying decisions.
What reliability actually requires
Start with consistency on the tasks you care about. Then calibrated confidence, meaning the system says when it’s unsure instead of guessing. That seems achievable: Anthropic research found that large models are often well calibrated about whether they know an answer. Reliability also needs verifiability, in the form of sources, reasoning or tests that let you check an answer quickly. And it needs graceful failure. When the system can’t do something, it should say so.
Notice that none of these require the model to be smarter. Several require it to be more honest.
A practical test Before relying on an AI tool for a recurring task, run it on twenty real examples and check every result. Your error rate on your own work is worth more than any published score.
What I want from AI companies
Publish error rates for common tasks, not just benchmark wins. Build interfaces that show sources and uncertainty. Reward models for saying "I don’t know". OpenAI researchers made this case in 2025, arguing that evaluations which penalize uncertainty encourage models to guess. Our guide to writing good prompts already recommends asking models to flag uncertainty; the products should make that the default.
Why this matters now
As AI moves from answering questions to taking actions, as agents do, reliability stops being a nice-to-have. An unreliable answer wastes a few minutes. An unreliable action can send the wrong email, delete the wrong file or pay the wrong invoice.
The next great leap in AI will not be the one that tops a leaderboard. It will be the one that makes these tools boringly, verifiably dependable, the kind of progress we praised in our column on boring AI.
Sources
- Benchmark Data Contamination of Large Language Models: A Survey, arXiv, 2024
- Language Models (Mostly) Know What They Know, arXiv, 2022
- Why language models hallucinate, OpenAI, September 2025
