Think about the professionals you trust most: a doctor, a mechanic, an accountant. Chances are they all have something in common. When they do not know something, they say so, and then they tell you how to find out.
Updated September 2026: we linked the OpenAI paper mentioned and added research on model calibration and sycophancy.
AI assistants are strikingly bad at this. Ask about something obscure and many will produce a fluent, specific, confident answer, whether or not it is right. That habit is the single biggest obstacle to trusting AI for work that matters.
Why AI guesses
Part of the reason is technical: language models generate plausible text, as we explain in our LLM explainer. But part of it is incentives. Training and evaluation have often rewarded answers that look complete and helpful. A model that says “I’m not sure” can score worse on a test than one that guesses, because a guess is sometimes right. OpenAI researchers made this argument in a 2025 paper on why language models hallucinate, suggesting that evaluations should stop penalizing expressions of uncertainty.
In other words, we have partly trained these systems to bluff. A 2023 study found a related habit: assistants tend to tell users what they want to hear, partly because people rate agreeable answers highly.
The cost of confident guessing
Every confident wrong answer teaches users one of two bad lessons. Either they learn to trust answers they should not, or they learn to distrust everything, including answers that are right. Both outcomes waste the real value of the technology. Our hallucinations guide exists because users currently have to do the calibration work themselves.
What honest AI would look like
- Clear uncertainty: “I’m confident about the first two points; I’m unsure about the date, and you should check it.”
- Sources by default for factual claims, with an easy way to open them
- Declining gracefully: “I don’t have reliable information on this company. Here’s where you could look.”
- Asking clarifying questions instead of assuming
- Stable answers: the same question should not yield confidently contradictory replies
Some of this is already happening. Anthropic research has found that large models are often well calibrated about whether they know an answer, so the raw material for honest uncertainty exists. Models have improved at admitting uncertainty, and search-connected assistants cite sources more consistently. But it is not yet the default experience.
What we can do as users
In the meantime, you can ask for honesty explicitly:
Add this to your custom instructions If you are not confident about a fact, say so plainly. Distinguish between what you know, what you are inferring and what you are guessing. Prefer “I don’t know” to a guess.
It will not fix the underlying incentives, but it helps, and our custom assistant guide explains how to make it stick.
A competitive advantage waiting to be claimed
The AI company that makes “I don’t know” a feature rather than a failure will earn something benchmarks cannot measure: trust. As we argued in our reliability column, dependable beats dazzling. An assistant that admits its limits is not less intelligent. It is more useful.
Sources
- Why language models hallucinate, OpenAI, September 2025
- Towards Understanding Sycophancy in Language Models, arXiv, 2023
- Language Models (Mostly) Know What They Know, arXiv, 2022
