A Confidence Score Is Not a Probability of Being Right

A claims manager put a file on the table: a storm claim on a home policy, with a repair estimate and photos. The system had read the documents, put the claim under the wrong type of cover and opened it with a reserve category, the label for money set aside to pay it, that no longer existed for that product. It went straight to a handler's queue, unreviewed. The log showed a confidence of 0.97. "If it was 97% sure," he asked, "why was it wrong?"

It was never 97% sure of anything. It was a score the model produced about its own output. It roughly tracked being right, the way a person's feeling of certainty does, but it wasn't a measured probability, and for this product, a month after the policy wording had changed, it wasn't even a useful hint.

That conversation ended our use of confidence cut-offs to decide which cases skip review. What replaced them is clumsier, and far easier to audit.

Fluent systems sound sure

None of this is new. In "On Calibration of Modern Neural Networks" (ICML 2017), Guo and colleagues showed that modern deep networks tend to be overconfident: reported confidence runs ahead of accuracy. Xiong and colleagues found at ICLR 2024 that language models asked for their confidence overstate it, bunching at high values whether right or not.

Inside an operation it gets worse. The cases a model gets wrong usually look perfectly normal: a familiar broker sending in a product that launched last week, or an estimate filed under a type of cover split in two last month. Nothing in the input says the world has changed, so the model's view of itself doesn't change either. Confident errors aren't a rare tail; they're how a fluent system fails when the business shifts quietly.

There's a fair objection. Anthropic's researchers showed in 2022, in "Language Models (Mostly) Know What They Know", that large models can be well calibrated on some question formats, meaning their stated confidence matches how often they're right, and that a simple adjustment removes much of the rest. The same paper found that this sense was poor on kinds of question the model hadn't been tuned on. That was our situation. You can adjust a score on last quarter's claims; the costly error arrives with next month's wording, before there's data to adjust on.

Skipping review is a permission per case type

So we replaced the cut-off with a permission per case type: a kind of claim, by channel and product line. A case type earns it on the insurer's own reviewed cases: a minimum run of consecutive cases checked by a person with no errors in the fields that move money (here, the type of cover and the reserve category), and every independent check passed on each case. The claims lead sets the run's length before go-live. It's recorded with its date, cases, and model and rule versions. Rare combinations are grouped, since on their own most would never collect enough cases.

A broker or product line the system hasn't seen before has no permission, so its first claims are always reviewed, however tidy they look. That alone removed a good share of our confident errors, because new products and forms are where silent misreads pile up.

It also changes the conversation. Nobody in claims wants to argue about calibration curves, but a manager can ask to see a permission, the cases behind it and what would take it away.

Evidence comes from checks that could have failed

The best evidence that an answer is right comes from checks that could fail on their own, not from the model. On a home claim they're mundane, which is their value. Was the policy in force on the date of loss? Is the proposed cover on this policy, and do the estimate's lines add up to its total? Is the reserve category valid for the product and claim type, and has it been used for this kind of loss before? All plain code, cheap to run, stored with the case.

With permission, a claim that clears them all goes through without anyone consulting the score. One failed check sends it to a handler, whatever the score. We kept the score for one job: it orders the review queue, lowest first, so the doubtful cases get read while people are fresh.

What an error costs overrides accuracy

A permission tells you how often a case type is right, not what a mistake costs. A glass claim with the wrong reserve category gets fixed at the next review; nobody outside the team notices. A claim sent down the property route while the documents mention an injury can miss a deadline that matters to the customer and to the insurer.

So some decisions never skip review, permission or not: any claim that mentions an injury, reserves above a limit the claims lead sets, claims that match the fraud team's indicators, and claims on policies that started only days before the loss. No message telling a customer their claim isn't covered goes out without a person. The system prepares these cases fully, and a person decides them.

The best case types need the closest watch

The errors that reach a settlement rarely come from messy case types. Those fail checks and get caught. The damage comes from case types that have been reliable for months. Reviewers stopped seeing them, and when a broker changes its form or a wording is revised, the model's view of itself doesn't move. Reliability removed the very observation that would have caught the change.

Stripe's primer on machine learning for fraud detection describes the mirror image: once a payment is blocked, nobody learns whether it was fraud, so the model's real accuracy there can only be estimated. Skipping review has the same blind spot: once a case goes through, nobody learns whether it was right until something breaks.

So every case type with permission keeps sending a small random sample to review, permanently. And four events withdraw a permission automatically: a confirmed error on a case that went through; any change to the model or the rules, a new policy wording included; a new form layout, detected by comparing document structure with known layouts; and claims arriving through a channel the permission wasn't earned on, such as a new broker portal. We kept the channel rule despite the friction: a new portal changes what documents look like more than anyone expects. When a permission lapses, nothing has failed. The system has noticed its evidence has gone stale.

Permissions are earned on current cases, never on a test set, which is a snapshot of the past. And the claims manager gets no single accuracy figure, which the biggest claim types would dominate, hiding where errors cluster. They get the case types with permission, the cases behind each, and every confident error since the last report.

One question settles it

When someone shows us a system that routes on model confidence, we ask one thing. Of the cases that went through unreviewed last month, how many turned out wrong, and of which types? Until that number exists, type by type, all anyone knows about the system is how sure it sounds.