A Confidence Score Is Not a Probability of Being Right

A claims manager put a file on the table. A storm claim on a home policy had come in with a contractor's repair estimate and a page of photos. The system had read the documents, put the claim under the wrong coverage line and opened it with a reserve category that no longer existed for that product. Nobody reviewed it; it went straight to a handler's queue. Next to the decision, the log showed a confidence of 0.97. "If it was 97% sure," he asked, "why was it wrong?"

It was never 97% sure of anything. The number was a score the model produced about its own output. Across thousands of claims it tracked being right, more or less, the way a person's feeling of certainty does. It was not a measured probability, and for this product, a month after the policy wording had changed, it wasn't even a useful hint.

That conversation ended our use of confidence thresholds for straight-through processing. What we use now is clumsier, and far easier to audit.

Fluent systems sound sure

None of this is new. In "On Calibration of Modern Neural Networks" (ICML 2017), Guo and colleagues showed that modern deep networks tend to be overconfident, with reported confidence running ahead of accuracy. Language models seem to go the same way. Xiong and colleagues found at ICLR 2024 that models asked to state their confidence tend to overstate it, and the answers bunch up at high values whether they are right or not.

Inside an operation it gets worse, and for a structural reason. The cases a model gets wrong usually look perfectly normal: a familiar broker sending in a product that launched last week, or a clean repair estimate filed under a coverage line that was split in two last month. Nothing in the input says the world has changed, so the model's view of itself doesn't change either. Confident errors aren't some rare tail. This is simply how a fluent system goes wrong when the business shifts quietly around it.

There is a fair objection, and we took it seriously. Anthropic's researchers showed in 2022, in "Language Models (Mostly) Know What They Know", that large models can be well calibrated on some question formats, and that a simple adjustment removes much of the miscalibration that is left. The same paper found the model's sense of whether it knew an answer was poorly calibrated on kinds of question it hadn't been tuned on. That was our situation exactly. You can recalibrate a score on last quarter's claims. The miscalibration that costs money turns up with next month's wording, before anyone has data to recalibrate on.

Straight-through is a licence per segment

So we stopped giving the system a threshold and started giving segments a licence. In claims intake a segment is a claim type, crossed with the channel the claim arrived through and the product line. A segment earns its licence on the insurer's own reviewed cases, and only when a minimum run of consecutive cases has been checked by a person with no errors in the fields that move money (here, the coverage line and the reserve category), and every independent check has passed on each case along the way. The claims lead agrees the length of that run before go-live. We write the licence into the case store with its date, the cases behind it and the model and rule versions in use. Low-volume combinations get grouped into one segment; on their own, most would never collect enough cases.

A broker or product line the system hasn't seen before has no licence, so its first claims are always reviewed, however tidy they look. That single consequence removed a good share of the confident errors we used to see, because new products and new forms are where silent misreads pile up.

It also changes the conversation. Nobody in a claims department wants to argue about calibration curves. A licence is something a manager can ask to see, together with the cases it was earned on and the events that would take it away.

Evidence comes from checks that could have failed

The best evidence that an answer is right rarely comes from the model that gave it. It comes from checks that could have failed on their own.

On a home claim they are mundane, which is part of their value. Does the policy number lead to a policy that was in force on the date of loss? Is the claimant the policyholder, or someone the policy names? We check that the proposed coverage line is actually on this policy, and that the repair estimate's lines add up to its total. Last, the reserve category has to be valid for the product and claim type, and to have been used for this kind of loss before. All of it is deterministic, cheap to run and stored against the case.

Inside a licensed segment, a claim that clears all of these goes through without anyone consulting the score. A single failed check puts it in front of a handler, and a 0.99 next to it changes nothing. We did keep the score for something in the end: it sets the order of the review queue, lowest first, so the doubtful cases get read while people are fresh.

Consequence overrides accuracy

A licence tells you how often a segment is right. It says nothing about what a mistake costs. If a small glass claim gets the wrong reserve category, the handler fixes it at the next review and nobody outside the team notices. If a claim goes down the property route while the documents mention an injury, it can miss a deadline that matters to the customer and to the insurer.

So some decisions never go straight through, licence or no licence. Any claim that mentions an injury. Reserves above a limit the claims lead sets. Claims that match the fraud team's indicators, and claims on policies that started only days before the loss. And nothing that would tell a customer their claim isn't covered is ever sent without a person. The system prepares these cases completely, and a person decides them.

The best segments need the closest watch

The errors that reach a settlement rarely come from messy segments. Messy claims fail checks, land in review and get caught. The damage comes from segments that have been reliable for months. They are licensed, so reviewers stopped seeing them, and when a broker changes its claim form or the product team revises a wording, the model's self-assessment doesn't move at all. Reliability removed the very observation that would have caught the change.

Stripe describes the mirror image in its primer on machine learning for fraud detection. Once a payment is blocked, nobody ever learns whether it was fraud, so the model's real accuracy above the threshold can only be estimated. Straight-through processing has the same blind spot on the other side: once a case goes through unreviewed, nobody learns whether it was right until something breaks.

We handle that in two ways. Every licensed segment keeps sending a small random sample to review, permanently. And four events revoke a licence automatically: a confirmed error on a case that went straight through; any change to the model or the classification rules, a new policy wording included; a new form layout, which we detect by comparing each document's structure with the layouts seen before; and claims of that type arriving through a channel the licence wasn't earned on, such as a new broker portal. We went back and forth on the channel rule, because it adds friction every time a new portal opens. We kept it. A new portal changes what the documents look like more than anyone expects. When a licence lapses, nothing has failed; the system has noticed its evidence has gone stale.

Licences are earned and kept on current production cases, never on the acceptance set, which is a snapshot of the past. The claims manager doesn't get one accuracy figure for the whole system, either, because that figure is dominated by the biggest claim types and hides the segments where errors cluster. What they get is the list of licensed segments, the cases behind each licence and every confident error since the last report.

One question settles it

When someone shows us a system that routes on model confidence, we ask one thing. Of the cases that went straight through last month, how many turned out to be wrong, and in which segments? Until that number exists, segment by segment, all anyone knows about the system is how sure it sounds.