A wrong answer that sounds right is the expensive one
A system that tells you it does not know is an inconvenience. A system that gives a wrong answer with the same confidence it shows when it is right is a liability.
What was observed
Teams evaluating a language model tend to track how often it succeeds. Far fewer track what its failures look like, and almost none track the property that decides the risk: whether the person receiving a failure can tell it apart from a success.
Those are different properties. A system that declines to answer when it is uncertain is annoying and safe. A system that answers every question in the same measured, well-organised register, right or wrong, is pleasant to use and unsafe in proportion to how much it is trusted.
The second is the common case, and it is the common case for a structural reason: fluency is what the training optimises, and correctness is what the reader infers from fluency.
What it means
In most settings a wrong answer produces an error someone catches. Where the answer gets acted on, it produces a commitment.
The cost of being wrong is not distributed evenly across industries. A marketing description that is subtly wrong is edited. A valuation that is subtly wrong is traded on, cleared, and settled, and by the time it is discovered the error has a counterparty. Fixed income and equity derivatives have this property structurally: an assumption that is confidently held and slightly wrong is not an embarrassment, it is a number somebody has already paid for.
The same structure appears anywhere a system’s output enters a process that cannot be taken back. An operational instruction to a plant. A reconciliation that closes a book. A control that a regulator will read back to you.
What to do next
Stop measuring only the success rate and start measuring the shape of the failures. Three questions are enough to begin, and none of them needs new tooling.
First, when the system is wrong, can the person receiving the output tell? If the answer is no, the system needs to express uncertainty in a form that survives being read quickly.
Second, is there a class of input where it is confidently wrong more often than average? That class is where the authority should be withdrawn, not where more prompt engineering should be applied.
Third, what does the system do at the edge of its competence: decline, hedge, or proceed? Only the first two are safe to build on, and which one it does is a design decision rather than a property to be discovered later.