A probability is only useful if it means what it says. Here is what calibration is, how to see it in a reliability diagram, why chat-trained models drift into overconfidence, what independent tests and our own measurements found for decision models, and how a few hundred labels let you measure and fix it.
Key takeaways
- Calibrated means: of all the answers given probability 0.8, about 80% turn out right.
- A reliability diagram plots stated probability against how often the model was right; the diagonal is perfect.
- Post-training for chat tends to make models overconfident; decision models are trained to keep probabilities honest, but that is a claim to test.
- Calibration is a property of each model on each task: in our tests Jev was slightly overconfident at the top, and on fresh news an LLM's stated confidence was better calibrated.
- Measure it on a few hundred of your own labels (ECE, Brier score), and fix systematic bias with simple recalibration such as Platt scaling.
Try it with your agent
Paste this into your coding agent:
For one decision we make with a model, collect 200-300 examples with trusted labels from real traffic. Run the model on all of them and log the probability of the chosen answer. Then report: accuracy, a 10-bin reliability table (stated vs actual), ECE and Brier score. Fit Platt scaling on half the data and report ECE on the other half, before and after. Save the script so we can rerun it on every model version change.
Transcript
Intro
Every answer from a decision model comes with probabilities. But a probability is only useful if it means what it says. This lesson is about checking that.
What calibration means
Think of a weather forecaster. On the days they say seventy percent chance of rain, it should rain on about seventy percent of them. That's calibration. It's not about any single day. It's about the group. Of all the answers a model gives at point eight, about eight in ten should be right.
Seeing it
You see calibration in a reliability diagram. Group answers by the probability the model gave, and plot how often each group was right. Perfect calibration is the diagonal. Here's Jev 1.13 on our 3,080 banking messages. Mostly close, but slightly below the line at the top: answers it gave about point nine nine were right ninety two percent of the time. A little overconfident.
Why chatbots drift
Why does this need saying at all? Because chat models are famously badly calibrated. Pretrained models are fairly well calibrated, but post-training for chat, as researcher Desi Ivanova puts it, destroys calibration. People like answers that sound sure, so models learn to sound sure.
Independent tests
Independent testers have checked the decision models directly. Ryan Porter ran 3,600 hard reasoning questions through OpenAI's Decisions API and Jev. Luna said ninety nine percent or more on 2,672 answers, and only sixty eight percent of those were right. Jev said ninety nine percent on 1,667 answers, and was right on ninety eight point nine.
Model by model
But don't turn that into a rule about paradigms. A team that checked Jev against 2,416 human judgments of police crash reports found it wasn't overconfident there at all. It hedged, staying too close to the middle. Their conclusion is the one to remember: calibration turns out to be a property of the individual model, rather than of the paradigm.
We saw the same thing on our fresh news test. Jev 1.14 was more accurate, but its top answers were a bit overconfident. The LLM, asked to write its confidence as a number, was actually better calibrated on this task. Calibration has to be measured, task by task.
Measuring it
Two numbers summarize it. Expected calibration error is the average gap between stated probability and actual accuracy, weighted by how many answers fall in each bin. Zero is perfect. The Brier score is the mean squared error of the probabilities themselves. Always guessing fifty fifty on a yes or no question scores point two five, so you want to be well below that.
You can measure it in a dozen lines. Bin the answers by stated probability, compare each bin's average probability with its accuracy, and weight the gaps. Run it on a few hundred of your own labeled examples, per question, and again whenever the model version changes.
Fixing it
And if it's off in a consistent direction, you can fix it cheaply. Fit a tiny logistic regression from the model's probability to what actually happened, called Platt scaling, on held-out labels. The crash narrative team cut calibration error by a factor of three point three this way. One limit: remapping fixes calibration, but it can't add information the probabilities don't carry.
Quick check
Quick check. A model gave probability point nine on five hundred answers, and four hundred of them were right. What does that tell you?
Try it
Try it. Have your agent build a calibration report for one of your decisions.
Recap
So: calibrated means point eight is right eight times in ten. See it in a reliability diagram, summarize it with ECE and Brier. Chat training drifts toward overconfidence, but every model has to be measured on every task. And a simple recalibration fixes the numbers. Next: turning probabilities into actions with thresholds.