{"id":"09-calibration","number":9,"slug":"calibration","title":"Calibration: does 0.9 mean 90%?","short":"Calibration","module":"Probabilities you can act on","summary":"A probability is only useful if it means what it says. Here is what calibration is, how to see it in a reliability diagram, why chat-trained models drift into overconfidence, what independent tests and our own measurements found for decision models, and how a few hundred labels let you measure and fix it.","voice":"Jessica","scenes":[{"template":"title","chapter":"Intro","narration":"Every answer from a decision model comes with probabilities. But a probability is only useful if it means what it says. This lesson is about checking that.","params":{"kicker":"Lesson 9","module":"Module 3 · Probabilities you can act on","title":"Calibration: does 0.9 *mean 90%*?","subtitle":"A probability is only useful if it means what it says.","cast":["snap","you"],"bubble":"0.9… right?","bubble_at":"means what it says"},"start":0.5,"dur":11.153,"words":[{"text":"Every","s":0.8,"e":1.12},{"text":"answer","s":1.2,"e":1.52},{"text":"from","s":1.6,"e":1.68},{"text":"a","s":1.76,"e":1.8},{"text":"decision","s":1.84,"e":2.24},{"text":"model","s":2.32,"e":2.64},{"text":"comes","s":2.72,"e":2.96},{"text":"with","s":3.04,"e":3.12},{"text":"probabilities.","s":3.2,"e":4.32},{"text":"But","s":4.56,"e":4.64},{"text":"a","s":4.68,"e":4.72},{"text":"probability","s":4.8,"e":5.44},{"text":"is","s":5.52,"e":5.76},{"text":"only","s":5.84,"e":6.08},{"text":"useful","s":6.16,"e":6.64},{"text":"if","s":6.72,"e":6.8},{"text":"it","s":6.88,"e":6.96},{"text":"means","s":7.12,"e":7.36},{"text":"what","s":7.44,"e":7.52},{"text":"it","s":7.6,"e":7.76},{"text":"says.","s":8,"e":8.64},{"text":"This","s":8.76,"e":8.88},{"text":"lesson","s":9.08,"e":9.28},{"text":"is","s":9.36,"e":9.44},{"text":"about","s":9.52,"e":9.68},{"text":"checking","s":9.84,"e":10.16},{"text":"that.","s":10.24,"e":10.8}]},{"template":"statement","chapter":"What calibration means","narration":"Think of a weather forecaster. On the days they say seventy percent chance of rain, it should rain on about seventy percent of them. That's calibration. It's not about any single day. It's about the group. Of all the answers a model gives at point eight, about eight in ten should be right.","params":{"eyebrow":"Calibration","text":"Of all the answers given *0.8*, about *8 in 10* should be right.","sub":"It's a property of groups of answers, never a promise about one.","sub_at":"not about any single day","art":"sun","tone":"agent","punch":"eight in ten"},"start":11.653,"dur":18.426,"words":[{"text":"Think","s":11.953,"e":12.193},{"text":"of","s":12.233,"e":12.273},{"text":"a","s":12.353,"e":12.393},{"text":"weather","s":12.433,"e":12.673},{"text":"forecaster.","s":12.753,"e":13.553},{"text":"On","s":13.953,"e":14.033},{"text":"the","s":14.073,"e":14.113},{"text":"days","s":14.233,"e":14.353},{"text":"they","s":14.433,"e":14.593},{"text":"say","s":14.753,"e":14.913},{"text":"seventy","s":15.153,"e":15.433},{"text":"percent","s":15.473,"e":15.793},{"text":"chance","s":15.873,"e":16.113},{"text":"of","s":16.153,"e":16.193},{"text":"rain,","s":16.273,"e":16.833},{"text":"it","s":16.913,"e":17.073},{"text":"should","s":17.193,"e":17.313},{"text":"rain","s":17.393,"e":17.633},{"text":"on","s":17.713,"e":17.793},{"text":"about","s":17.993,"e":18.193},{"text":"seventy","s":18.353,"e":18.633},{"text":"percent","s":18.673,"e":18.993},{"text":"of","s":19.033,"e":19.073},{"text":"them.","s":19.153,"e":19.633},{"text":"That's","s":19.873,"e":20.193},{"text":"calibration.","s":20.273,"e":21.233},{"text":"It's","s":21.393,"e":21.553},{"text":"not","s":21.633,"e":21.713},{"text":"about","s":21.833,"e":21.953},{"text":"any","s":22.073,"e":22.193},{"text":"single","s":22.353,"e":22.673},{"text":"day.","s":22.753,"e":23.193},{"text":"It's","s":23.313,"e":23.433},{"text":"about","s":23.473,"e":23.633},{"text":"the","s":23.673,"e":23.713},{"text":"group.","s":23.793,"e":24.273},{"text":"Of","s":24.673,"e":24.913},{"text":"all","s":24.993,"e":25.073},{"text":"the","s":25.153,"e":25.233},{"text":"answers","s":25.313,"e":25.633},{"text":"a","s":25.673,"e":25.713},{"text":"model","s":25.793,"e":26.033},{"text":"gives","s":26.113,"e":26.273},{"text":"at","s":26.353,"e":26.433},{"text":"point","s":26.513,"e":26.673},{"text":"eight,","s":26.833,"e":27.313},{"text":"about","s":27.513,"e":27.713},{"text":"eight","s":27.793,"e":27.873},{"text":"in","s":27.953,"e":28.193},{"text":"ten","s":28.353,"e":28.513},{"text":"should","s":28.593,"e":28.753},{"text":"be","s":28.793,"e":28.833},{"text":"right.","s":28.913,"e":29.473}]},{"template":"reliability","chapter":"Seeing it","narration":"You see calibration in a reliability diagram. Group answers by the probability the model gave, and plot how often each group was right. Perfect calibration is the diagonal. Here's Jev 1.13 on our 3,080 banking messages. Mostly close, but slightly below the line at the top: answers it gave about point nine nine were right ninety two percent of the time. A little overconfident.","params":{"title":"Jev 1.13 on BANKING77","series":[{"name":"Jev 1.13 · 3,080 messages","tone":"s1","stat":"ECE 0.089","gaps":true,"at":"Here's Jev","bins":[{"conf":0.345,"acc":0.367,"n":30},{"conf":0.452,"acc":0.367,"n":79},{"conf":0.541,"acc":0.478,"n":186},{"conf":0.645,"acc":0.542,"n":190},{"conf":0.746,"acc":0.497,"n":195},{"conf":0.848,"acc":0.679,"n":280},{"conf":0.986,"acc":0.92,"n":2117}]}],"diag_at":"Perfect calibration","note":"Dots below the line: more confident than right. Dot size = number of answers.","note_at":"A little overconfident"},"start":30.079,"dur":26.498,"words":[{"text":"You","s":30.379,"e":30.512},{"text":"see","s":30.539,"e":30.726},{"text":"calibration","s":30.779,"e":31.429},{"text":"in","s":31.499,"e":31.579},{"text":"a","s":31.659,"e":31.699},{"text":"reliability","s":31.739,"e":32.379},{"text":"diagram.","s":32.459,"e":33.339},{"text":"Group","s":33.579,"e":33.819},{"text":"answers","s":33.899,"e":34.299},{"text":"by","s":34.379,"e":34.459},{"text":"the","s":34.499,"e":34.539},{"text":"probability","s":34.619,"e":35.259},{"text":"the","s":35.299,"e":35.339},{"text":"model","s":35.419,"e":35.659},{"text":"gave,","s":35.819,"e":36.379},{"text":"and","s":36.499,"e":36.619},{"text":"plot","s":36.699,"e":36.939},{"text":"how","s":37.059,"e":37.179},{"text":"often","s":37.259,"e":37.499},{"text":"each","s":37.579,"e":37.739},{"text":"group","s":37.819,"e":37.979},{"text":"was","s":38.059,"e":38.139},{"text":"right.","s":38.219,"e":38.779},{"text":"Perfect","s":39.019,"e":39.339},{"text":"calibration","s":39.419,"e":40.139},{"text":"is","s":40.219,"e":40.299},{"text":"the","s":40.379,"e":40.419},{"text":"diagonal.","s":40.459,"e":41.419},{"text":"Here's","s":41.739,"e":41.979},{"text":"Jev","s":42.059,"e":42.379},{"text":"1.13","s":42.459,"e":42.939},{"text":"on","s":42.979,"e":43.006},{"text":"our","s":43.179,"e":43.819},{"text":"3,080","s":43.899,"e":44.699},{"text":"banking","s":44.779,"e":45.099},{"text":"messages.","s":45.419,"e":45.979},{"text":"Mostly","s":46.219,"e":46.539},{"text":"close,","s":46.619,"e":47.179},{"text":"but","s":47.299,"e":47.419},{"text":"slightly","s":47.579,"e":48.059},{"text":"below","s":48.139,"e":48.379},{"text":"the","s":48.419,"e":48.459},{"text":"line","s":48.539,"e":48.699},{"text":"at","s":48.739,"e":48.779},{"text":"the","s":48.859,"e":48.939},{"text":"top:","s":49.019,"e":49.739},{"text":"answers","s":49.899,"e":50.059},{"text":"it","s":50.139,"e":50.219},{"text":"gave","s":50.259,"e":50.459},{"text":"about","s":50.619,"e":50.779},{"text":"point","s":50.859,"e":51.019},{"text":"nine","s":51.179,"e":51.419},{"text":"nine","s":51.499,"e":51.739},{"text":"were","s":51.899,"e":52.059},{"text":"right","s":52.179,"e":52.299},{"text":"ninety","s":52.379,"e":52.699},{"text":"two","s":52.859,"e":52.939},{"text":"percent","s":53.099,"e":53.339},{"text":"of","s":53.379,"e":53.419},{"text":"the","s":53.499,"e":53.579},{"text":"time.","s":53.739,"e":54.299},{"text":"A","s":54.379,"e":54.459},{"text":"little","s":54.539,"e":54.779},{"text":"overconfident.","s":54.859,"e":55.979}]},{"template":"quote","chapter":"Why chatbots drift","narration":"Why does this need saying at all? Because chat models are famously badly calibrated. Pretrained models are fairly well calibrated, but post-training for chat, as researcher Desi Ivanova puts it, destroys calibration. People like answers that sound sure, so models learn to sound sure.","params":{"text":"Post training… the losses there are not proper scoring rules… *actually destroys calibration*, and this is also very well documented.","who":"Desi Ivanova","role":"researcher, on LLM uncertainty","src":"Estimating and Calibrating Uncertainty in LLMs","t":1175},"start":56.577,"dur":20.281,"words":[{"text":"Why","s":56.877,"e":57.117},{"text":"does","s":57.197,"e":57.357},{"text":"this","s":57.437,"e":57.517},{"text":"need","s":57.637,"e":57.757},{"text":"saying","s":57.837,"e":58.077},{"text":"at","s":58.157,"e":58.237},{"text":"all?","s":58.437,"e":58.877},{"text":"Because","s":59.117,"e":59.357},{"text":"chat","s":59.437,"e":59.677},{"text":"models","s":59.757,"e":60.077},{"text":"are","s":60.157,"e":60.397},{"text":"famously","s":60.477,"e":61.037},{"text":"badly","s":61.197,"e":61.597},{"text":"calibrated.","s":61.717,"e":62.717},{"text":"Pretrained","s":62.837,"e":63.197},{"text":"models","s":63.277,"e":63.597},{"text":"are","s":63.717,"e":63.837},{"text":"fairly","s":63.957,"e":64.237},{"text":"well","s":64.317,"e":64.397},{"text":"calibrated,","s":64.477,"e":65.437},{"text":"but","s":65.557,"e":65.677},{"text":"post-training","s":65.757,"e":66.317},{"text":"for","s":66.397,"e":66.477},{"text":"chat,","s":66.557,"e":67.117},{"text":"as","s":67.237,"e":67.357},{"text":"researcher","s":67.557,"e":67.837},{"text":"Desi","s":67.917,"e":68.157},{"text":"Ivanova","s":68.237,"e":68.717},{"text":"puts","s":68.837,"e":68.957},{"text":"it,","s":69.037,"e":69.437},{"text":"destroys","s":69.637,"e":70.237},{"text":"calibration.","s":70.317,"e":71.597},{"text":"People","s":71.757,"e":71.917},{"text":"like","s":72.037,"e":72.157},{"text":"answers","s":72.357,"e":72.557},{"text":"that","s":72.637,"e":72.717},{"text":"sound","s":72.797,"e":73.037},{"text":"sure,","s":73.197,"e":73.917},{"text":"so","s":74.077,"e":74.237},{"text":"models","s":74.477,"e":74.797},{"text":"learn","s":74.877,"e":75.117},{"text":"to","s":75.197,"e":75.277},{"text":"sound","s":75.397,"e":75.517},{"text":"sure.","s":75.597,"e":76.237}]},{"template":"compare","chapter":"Independent tests","narration":"Independent testers have checked the decision models directly. Ryan Porter ran 3,600 hard reasoning questions through OpenAI's Decisions API and Jev. Luna said ninety nine percent or more on 2,672 answers, and only sixty eight percent of those were right. Jev said ninety nine percent on 1,667 answers, and was right on ninety eight point nine.","params":{"title":"Answers stated at ≥ 99% (3,600 hard questions)","left":{"title":"Decisions API (Luna)","who":"mull","tone":"bad","items":[{"text":"**2,672** answers at ≥ 99%","at":"2,672"},{"text":"**68%** of them right","at":"sixty eight"}]},"right":{"title":"Jev","who":"snap","tone":"good","items":[{"text":"**1,667** answers at ≥ 99%","at":"1,667"},{"text":"**98.9%** of them right","at":"ninety eight point nine"}]},"vs":"vs","hnote":{"text":"Ryan Porter, Anthus, Oct 2026","at":"Ryan Porter","p":[50,77],"pp":[50,76]}},"start":76.858,"dur":27.229,"words":[{"text":"Independent","s":77.158,"e":77.718},{"text":"testers","s":77.878,"e":78.198},{"text":"have","s":78.278,"e":78.438},{"text":"checked","s":78.558,"e":78.678},{"text":"the","s":78.718,"e":78.758},{"text":"decision","s":78.958,"e":79.158},{"text":"models","s":79.238,"e":79.558},{"text":"directly.","s":79.638,"e":80.758},{"text":"Ryan","s":80.838,"e":81.078},{"text":"Porter","s":81.158,"e":81.478},{"text":"ran","s":81.558,"e":81.798},{"text":"3,600","s":81.878,"e":82.038},{"text":"hard","s":82.198,"e":83.238},{"text":"reasoning","s":83.398,"e":84.038},{"text":"questions","s":84.118,"e":84.598},{"text":"through","s":84.678,"e":84.918},{"text":"OpenAI's","s":85.078,"e":85.558},{"text":"Decisions","s":85.638,"e":86.118},{"text":"API","s":86.158,"e":86.758},{"text":"and","s":86.838,"e":86.918},{"text":"Jev.","s":87.078,"e":88.038},{"text":"Luna","s":88.118,"e":88.358},{"text":"said","s":88.438,"e":88.678},{"text":"ninety","s":88.758,"e":88.998},{"text":"nine","s":89.078,"e":89.238},{"text":"percent","s":89.318,"e":89.638},{"text":"or","s":89.718,"e":89.798},{"text":"more","s":89.958,"e":90.278},{"text":"on","s":90.358,"e":90.598},{"text":"2,672","s":90.678,"e":91.158},{"text":"answers,","s":91.318,"e":93.478},{"text":"and","s":93.518,"e":93.558},{"text":"only","s":93.638,"e":93.878},{"text":"sixty","s":93.958,"e":94.358},{"text":"eight","s":94.398,"e":94.438},{"text":"percent","s":94.598,"e":94.838},{"text":"of","s":94.878,"e":94.918},{"text":"those","s":94.998,"e":95.158},{"text":"were","s":95.238,"e":95.318},{"text":"right.","s":95.398,"e":96.278},{"text":"Jev","s":96.358,"e":96.598},{"text":"said","s":96.678,"e":96.918},{"text":"ninety","s":96.998,"e":97.318},{"text":"nine","s":97.398,"e":97.478},{"text":"percent","s":97.558,"e":97.958},{"text":"on","s":98.038,"e":98.198},{"text":"1,667","s":98.278,"e":100.198},{"text":"answers,","s":100.278,"e":101.078},{"text":"and","s":101.118,"e":101.158},{"text":"was","s":101.238,"e":101.398},{"text":"right","s":101.518,"e":101.638},{"text":"on","s":101.718,"e":101.958},{"text":"ninety","s":102.038,"e":102.278},{"text":"eight","s":102.358,"e":102.438},{"text":"point","s":102.518,"e":102.758},{"text":"nine.","s":102.838,"e":103.478}]},{"template":"quote","chapter":"Model by model","narration":"But don't turn that into a rule about paradigms. A team that checked Jev against 2,416 human judgments of police crash reports found it wasn't overconfident there at all. It hedged, staying too close to the middle. Their conclusion is the one to remember: calibration turns out to be a property of the individual model, rather than of the paradigm.","params":{"text":"Calibration turns out to be *a property of the individual model* rather than of the paradigm.","who":"Amir Rafe & Subasish Das","role":"Calibrated Decisions at Scale (arXiv)","src":"police crash narratives with Jev, 2026"},"start":104.087,"dur":24.016,"words":[{"text":"But","s":104.387,"e":104.627},{"text":"don't","s":104.707,"e":104.867},{"text":"turn","s":104.987,"e":105.107},{"text":"that","s":105.187,"e":105.267},{"text":"into","s":105.387,"e":105.507},{"text":"a","s":105.587,"e":105.747},{"text":"rule","s":105.827,"e":106.147},{"text":"about","s":106.307,"e":106.467},{"text":"paradigms.","s":106.587,"e":107.747},{"text":"A","s":107.827,"e":107.907},{"text":"team","s":108.027,"e":108.147},{"text":"that","s":108.227,"e":108.307},{"text":"checked","s":108.427,"e":108.547},{"text":"Jev","s":108.627,"e":108.867},{"text":"against","s":109.067,"e":109.267},{"text":"2,416","s":109.347,"e":111.267},{"text":"human","s":111.427,"e":111.747},{"text":"judgments","s":111.827,"e":112.307},{"text":"of","s":112.387,"e":112.467},{"text":"police","s":112.547,"e":112.787},{"text":"crash","s":112.867,"e":113.107},{"text":"reports","s":113.507,"e":113.907},{"text":"found","s":114.067,"e":114.227},{"text":"it","s":114.307,"e":114.387},{"text":"wasn't","s":114.467,"e":114.707},{"text":"overconfident","s":114.787,"e":115.507},{"text":"there","s":115.587,"e":115.667},{"text":"at","s":115.707,"e":115.747},{"text":"all.","s":115.827,"e":116.547},{"text":"It","s":116.627,"e":116.707},{"text":"hedged,","s":116.787,"e":117.507},{"text":"staying","s":117.547,"e":117.907},{"text":"too","s":117.987,"e":118.147},{"text":"close","s":118.227,"e":118.467},{"text":"to","s":118.507,"e":118.547},{"text":"the","s":118.627,"e":118.707},{"text":"middle.","s":118.867,"e":119.507},{"text":"Their","s":119.627,"e":119.747},{"text":"conclusion","s":119.867,"e":120.467},{"text":"is","s":120.547,"e":120.627},{"text":"the","s":120.667,"e":120.707},{"text":"one","s":120.787,"e":120.867},{"text":"to","s":120.947,"e":121.027},{"text":"remember:","s":121.267,"e":121.987},{"text":"calibration","s":122.067,"e":122.787},{"text":"turns","s":122.907,"e":123.107},{"text":"out","s":123.187,"e":123.267},{"text":"to","s":123.307,"e":123.347},{"text":"be","s":123.427,"e":123.467},{"text":"a","s":123.507,"e":123.587},{"text":"property","s":123.827,"e":124.067},{"text":"of","s":124.147,"e":124.227},{"text":"the","s":124.307,"e":124.387},{"text":"individual","s":124.467,"e":125.107},{"text":"model,","s":125.307,"e":125.827},{"text":"rather","s":125.987,"e":126.147},{"text":"than","s":126.227,"e":126.387},{"text":"of","s":126.427,"e":126.467},{"text":"the","s":126.547,"e":126.627},{"text":"paradigm.","s":126.707,"e":127.507}]},{"template":"reliability","chapter":"Model by model","narration":"We saw the same thing on our fresh news test. Jev 1.14 was more accurate, but its top answers were a bit overconfident. The LLM, asked to write its confidence as a number, was actually better calibrated on this task. Calibration has to be measured, task by task.","params":{"title":"Fresh news, 279 stories","series":[{"name":"Jev 1.14","tone":"s1","stat":"81.7% right · ECE 0.10","at":"Jev 1.14","bins":[{"conf":0.547,"acc":0.529,"n":17},{"conf":0.644,"acc":0.846,"n":13},{"conf":0.75,"acc":0.652,"n":23},{"conf":0.857,"acc":0.692,"n":26},{"conf":0.984,"acc":0.89,"n":191}]},{"name":"Haiku 5.5, stated","tone":"s2","stat":"79.6% right · ECE 0.065","at":"The LLM","bins":[{"conf":0.62,"acc":0.444,"n":27},{"conf":0.747,"acc":0.758,"n":33},{"conf":0.85,"acc":0.735,"n":102},{"conf":0.949,"acc":0.948,"n":115}]}],"note":"Measured for this course, 2026-10-15. Bins with fewer than 10 answers left out.","note_at":"task by task"},"start":128.103,"dur":19.863,"words":[{"text":"We","s":128.403,"e":128.563},{"text":"saw","s":128.616,"e":128.723},{"text":"the","s":128.803,"e":128.883},{"text":"same","s":128.963,"e":129.203},{"text":"thing","s":129.283,"e":129.443},{"text":"on","s":129.523,"e":129.603},{"text":"our","s":129.683,"e":129.763},{"text":"fresh","s":129.883,"e":130.003},{"text":"news","s":130.083,"e":130.243},{"text":"test.","s":130.323,"e":131.203},{"text":"Jev","s":131.283,"e":131.523},{"text":"1.14","s":131.603,"e":132.643},{"text":"was","s":132.763,"e":132.883},{"text":"more","s":133.003,"e":133.123},{"text":"accurate,","s":133.203,"e":133.923},{"text":"but","s":134.043,"e":134.163},{"text":"its","s":134.243,"e":134.483},{"text":"top","s":134.643,"e":134.803},{"text":"answers","s":134.883,"e":135.283},{"text":"were","s":135.363,"e":135.443},{"text":"a","s":135.563,"e":135.683},{"text":"bit","s":135.763,"e":136.003},{"text":"overconfident.","s":136.163,"e":137.363},{"text":"The","s":137.483,"e":137.603},{"text":"LLM,","s":137.683,"e":138.643},{"text":"asked","s":138.763,"e":138.963},{"text":"to","s":139.003,"e":139.043},{"text":"write","s":139.123,"e":139.203},{"text":"its","s":139.283,"e":139.443},{"text":"confidence","s":139.683,"e":139.923},{"text":"as","s":140.003,"e":140.083},{"text":"a","s":140.136,"e":140.19},{"text":"number,","s":140.243,"e":140.963},{"text":"was","s":141.083,"e":141.203},{"text":"actually","s":141.283,"e":141.683},{"text":"better","s":141.843,"e":142.003},{"text":"calibrated","s":142.083,"e":142.643},{"text":"on","s":142.683,"e":142.723},{"text":"this","s":142.803,"e":142.963},{"text":"task.","s":143.203,"e":143.843},{"text":"Calibration","s":143.923,"e":144.563},{"text":"has","s":144.723,"e":144.963},{"text":"to","s":145.003,"e":145.043},{"text":"be","s":145.123,"e":145.203},{"text":"measured,","s":145.283,"e":145.923},{"text":"task","s":146.123,"e":146.323},{"text":"by","s":146.483,"e":146.563},{"text":"task.","s":146.643,"e":147.363}]},{"template":"formula","chapter":"Measuring it","narration":"Two numbers summarize it. Expected calibration error is the average gap between stated probability and actual accuracy, weighted by how many answers fall in each bin. Zero is perfect. The Brier score is the mean squared error of the probabilities themselves. Always guessing fifty fifty on a yes or no question scores point two five, so you want to be well below that.","params":{"title":"Two numbers to track","parts":[{"text":"ECE","sub":"weighted gap between stated and actual · 0 is perfect","tone":"agent","at":"Expected calibration error"},{"op":"+"},{"text":"Brier","sub":"mean squared error of the probabilities · coin flip = 0.25","tone":"s2","at":"Brier score"}],"caption":"Track both, per question, on a few hundred labeled examples from your own traffic.","caption_at":"well below"},"start":147.966,"dur":25.139,"words":[{"text":"Two","s":148.266,"e":148.506},{"text":"numbers","s":148.586,"e":148.906},{"text":"summarize","s":148.986,"e":149.386},{"text":"it.","s":149.466,"e":149.706},{"text":"Expected","s":150.106,"e":150.586},{"text":"calibration","s":150.666,"e":151.226},{"text":"error","s":151.306,"e":151.626},{"text":"is","s":151.706,"e":151.866},{"text":"the","s":151.986,"e":152.106},{"text":"average","s":152.186,"e":152.506},{"text":"gap","s":152.586,"e":152.826},{"text":"between","s":152.906,"e":153.146},{"text":"stated","s":153.306,"e":153.626},{"text":"probability","s":153.706,"e":154.346},{"text":"and","s":154.426,"e":154.586},{"text":"actual","s":154.746,"e":155.146},{"text":"accuracy,","s":155.226,"e":156.266},{"text":"weighted","s":156.426,"e":156.586},{"text":"by","s":156.666,"e":156.826},{"text":"how","s":156.906,"e":156.986},{"text":"many","s":157.066,"e":157.226},{"text":"answers","s":157.306,"e":157.706},{"text":"fall","s":157.786,"e":157.946},{"text":"in","s":157.986,"e":158.026},{"text":"each","s":158.106,"e":158.266},{"text":"bin.","s":158.426,"e":158.746},{"text":"Zero","s":159.306,"e":159.706},{"text":"is","s":159.866,"e":159.946},{"text":"perfect.","s":160.106,"e":160.666},{"text":"The","s":161.146,"e":161.226},{"text":"Brier","s":161.306,"e":161.626},{"text":"score","s":161.706,"e":162.026},{"text":"is","s":162.106,"e":162.186},{"text":"the","s":162.266,"e":162.346},{"text":"mean","s":162.506,"e":162.666},{"text":"squared","s":162.746,"e":163.146},{"text":"error","s":163.226,"e":163.466},{"text":"of","s":163.546,"e":163.626},{"text":"the","s":163.666,"e":163.706},{"text":"probabilities","s":163.786,"e":164.426},{"text":"themselves.","s":164.466,"e":165.466},{"text":"Always","s":165.706,"e":165.946},{"text":"guessing","s":166.026,"e":166.346},{"text":"fifty","s":166.426,"e":166.666},{"text":"fifty","s":166.826,"e":167.146},{"text":"on","s":167.226,"e":167.306},{"text":"a","s":167.386,"e":167.426},{"text":"yes","s":167.466,"e":167.626},{"text":"or","s":167.706,"e":167.786},{"text":"no","s":167.866,"e":167.946},{"text":"question","s":168.026,"e":168.506},{"text":"scores","s":168.586,"e":168.986},{"text":"point","s":169.146,"e":169.306},{"text":"two","s":169.426,"e":169.546},{"text":"five,","s":169.746,"e":170.266},{"text":"so","s":170.346,"e":170.506},{"text":"you","s":170.666,"e":170.746},{"text":"want","s":170.826,"e":170.906},{"text":"to","s":170.946,"e":170.986},{"text":"be","s":171.066,"e":171.146},{"text":"well","s":171.306,"e":171.546},{"text":"below","s":171.626,"e":171.866},{"text":"that.","s":171.946,"e":172.506}]},{"template":"code","chapter":"Measuring it","narration":"You can measure it in a dozen lines. Bin the answers by stated probability, compare each bin's average probability with its accuracy, and weight the gaps. Run it on a few hundred of your own labeled examples, per question, and again whenever the model version changes.","params":{"file":"calibration.py","lang":"py","lines":["def ece(rows, bins=10):","    # rows: (stated_probability, was_right)","    groups = [[] for _ in range(bins)]","    for p, ok in rows:","        groups[min(int(p * bins), bins - 1)].append((p, ok))","    err = 0.0","    for grp in groups:","        if grp:","            conf = sum(p for p, _ in grp) / len(grp)","            acc = sum(ok for _, ok in grp) / len(grp)","            err += len(grp) / len(rows) * abs(conf - acc)","    return err","","brier = sum((p - ok) ** 2 for p, ok in rows) / len(rows)"],"hl":[{"from":9,"to":11,"label":"stated vs actual, weighted","tone":"agent","at":"compare each bin"}]},"start":173.105,"dur":17.146,"words":[{"text":"You","s":173.405,"e":173.565},{"text":"can","s":173.645,"e":173.725},{"text":"measure","s":173.885,"e":174.045},{"text":"it","s":174.125,"e":174.205},{"text":"in","s":174.245,"e":174.285},{"text":"a","s":174.365,"e":174.525},{"text":"dozen","s":174.565,"e":174.845},{"text":"lines.","s":174.925,"e":175.605},{"text":"Bin","s":175.805,"e":175.965},{"text":"the","s":176.005,"e":176.045},{"text":"answers","s":176.125,"e":176.525},{"text":"by","s":176.605,"e":176.685},{"text":"stated","s":176.765,"e":177.085},{"text":"probability,","s":177.445,"e":178.125},{"text":"compare","s":178.205,"e":178.605},{"text":"each","s":178.725,"e":178.845},{"text":"bin's","s":178.925,"e":179.325},{"text":"average","s":179.485,"e":179.645},{"text":"probability","s":179.725,"e":180.365},{"text":"with","s":180.405,"e":180.445},{"text":"its","s":180.565,"e":180.685},{"text":"accuracy,","s":180.765,"e":181.645},{"text":"and","s":181.725,"e":181.805},{"text":"weight","s":181.925,"e":182.045},{"text":"the","s":182.085,"e":182.125},{"text":"gaps.","s":182.205,"e":183.005},{"text":"Run","s":183.085,"e":183.165},{"text":"it","s":183.245,"e":183.325},{"text":"on","s":183.365,"e":183.405},{"text":"a","s":183.485,"e":183.645},{"text":"few","s":183.725,"e":183.805},{"text":"hundred","s":183.885,"e":184.205},{"text":"of","s":184.285,"e":184.365},{"text":"your","s":184.445,"e":184.525},{"text":"own","s":184.605,"e":184.765},{"text":"labeled","s":184.805,"e":185.085},{"text":"examples,","s":185.405,"e":185.885},{"text":"per","s":185.965,"e":186.045},{"text":"question,","s":186.125,"e":186.925},{"text":"and","s":187.005,"e":187.085},{"text":"again","s":187.285,"e":187.485},{"text":"whenever","s":187.565,"e":187.965},{"text":"the","s":188.045,"e":188.085},{"text":"model","s":188.125,"e":188.365},{"text":"version","s":188.445,"e":188.685},{"text":"changes.","s":188.765,"e":189.645}]},{"template":"statement","chapter":"Fixing it","narration":"And if it's off in a consistent direction, you can fix it cheaply. Fit a tiny logistic regression from the model's probability to what actually happened, called Platt scaling, on held-out labels. The crash narrative team cut calibration error by a factor of three point three this way. One limit: remapping fixes calibration, but it can't add information the probabilities don't carry.","params":{"eyebrow":"Platt scaling: a 2-parameter fix","text":"Recalibration fixes the *numbers*, not the *ranking*.","sub":"Crash-narrative study: ECE 0.0231 → 0.0069 (3.3×) with a few thousand labels.","sub_at":"three point three","art":"tool","tone":"good","punch":"three point three"},"start":190.251,"dur":24.434,"words":[{"text":"And","s":190.551,"e":190.711},{"text":"if","s":190.751,"e":190.791},{"text":"it's","s":190.871,"e":191.031},{"text":"off","s":191.151,"e":191.271},{"text":"in","s":191.311,"e":191.351},{"text":"a","s":191.391,"e":191.431},{"text":"consistent","s":191.511,"e":191.991},{"text":"direction,","s":192.071,"e":192.871},{"text":"you","s":192.911,"e":192.951},{"text":"can","s":193.031,"e":193.111},{"text":"fix","s":193.191,"e":193.351},{"text":"it","s":193.431,"e":193.511},{"text":"cheaply.","s":193.631,"e":194.271},{"text":"Fit","s":194.551,"e":194.711},{"text":"a","s":194.791,"e":194.871},{"text":"tiny","s":195.071,"e":195.271},{"text":"logistic","s":195.391,"e":195.751},{"text":"regression","s":195.831,"e":196.311},{"text":"from","s":196.391,"e":196.551},{"text":"the","s":196.591,"e":196.631},{"text":"model's","s":196.711,"e":197.031},{"text":"probability","s":197.391,"e":197.751},{"text":"to","s":197.791,"e":197.831},{"text":"what","s":197.911,"e":198.071},{"text":"actually","s":198.151,"e":198.551},{"text":"happened,","s":198.631,"e":199.271},{"text":"called","s":199.391,"e":199.511},{"text":"Platt","s":199.671,"e":199.911},{"text":"scaling,","s":199.991,"e":200.631},{"text":"on","s":200.711,"e":200.791},{"text":"held-out","s":200.871,"e":201.191},{"text":"labels.","s":201.271,"e":202.231},{"text":"The","s":202.271,"e":202.311},{"text":"crash","s":202.391,"e":202.631},{"text":"narrative","s":202.711,"e":203.031},{"text":"team","s":203.191,"e":203.351},{"text":"cut","s":203.471,"e":203.591},{"text":"calibration","s":203.671,"e":204.231},{"text":"error","s":204.311,"e":204.551},{"text":"by","s":204.631,"e":204.711},{"text":"a","s":204.791,"e":204.871},{"text":"factor","s":205.031,"e":205.191},{"text":"of","s":205.271,"e":205.431},{"text":"three","s":205.511,"e":205.671},{"text":"point","s":205.831,"e":206.151},{"text":"three","s":206.231,"e":206.311},{"text":"this","s":206.471,"e":206.711},{"text":"way.","s":206.831,"e":207.511},{"text":"One","s":207.631,"e":207.751},{"text":"limit:","s":207.951,"e":208.551},{"text":"remapping","s":208.631,"e":209.111},{"text":"fixes","s":209.271,"e":209.591},{"text":"calibration,","s":209.671,"e":210.551},{"text":"but","s":210.671,"e":210.791},{"text":"it","s":210.871,"e":211.031},{"text":"can't","s":211.111,"e":211.351},{"text":"add","s":211.431,"e":211.511},{"text":"information","s":211.791,"e":212.071},{"text":"the","s":212.151,"e":212.231},{"text":"probabilities","s":212.311,"e":212.951},{"text":"don't","s":213.031,"e":213.271},{"text":"carry.","s":213.511,"e":214.071}]},{"template":"quiz","chapter":"Quick check","narration":"Quick check. A model gave probability point nine on five hundred answers, and four hundred of them were right. What does that tell you?","params":{"q":"500 answers stated at **0.9**; **400** were right. What does that tell you?","options":["The model is well calibrated","It's overconfident at 0.9: those answers are right about 80% of the time","It's underconfident","Nothing: calibration is about single answers"],"answer":1,"explain":"400 of 500 is 80%, below the 90% it claimed. Recalibrate, or treat its 0.9 as 0.8 when you set thresholds."},"start":214.685,"dur":13.859,"words":[{"text":"Quick","s":214.985,"e":215.225},{"text":"check.","s":215.305,"e":215.825},{"text":"A","s":216.025,"e":216.105},{"text":"model","s":216.185,"e":216.425},{"text":"gave","s":216.505,"e":216.665},{"text":"probability","s":216.745,"e":217.385},{"text":"point","s":217.545,"e":217.785},{"text":"nine","s":217.945,"e":218.265},{"text":"on","s":218.425,"e":218.665},{"text":"five","s":218.785,"e":218.905},{"text":"hundred","s":218.985,"e":219.225},{"text":"answers,","s":219.505,"e":219.945},{"text":"and","s":220.065,"e":220.185},{"text":"four","s":220.345,"e":220.505},{"text":"hundred","s":220.585,"e":220.825},{"text":"of","s":220.865,"e":220.905},{"text":"them","s":220.985,"e":221.065},{"text":"were","s":221.145,"e":221.305},{"text":"right.","s":221.505,"e":221.945},{"text":"What","s":222.185,"e":222.265},{"text":"does","s":222.345,"e":222.425},{"text":"that","s":222.505,"e":222.585},{"text":"tell","s":222.665,"e":222.825},{"text":"you?","s":222.905,"e":223.385}]},{"template":"exercise","chapter":"Try it","narration":"Try it. Have your agent build a calibration report for one of your decisions.","params":{"title":"Measure your calibration","intro":"Paste this into your coding agent:","prompt":"For one decision we make with a model, collect 200-300 examples with\ntrusted labels from real traffic. Run the model on all of them and log\nthe probability of the chosen answer. Then report: accuracy, a\n10-bin reliability table (stated vs actual), ECE and Brier score.\nFit Platt scaling on half the data and report ECE on the other half,\nbefore and after. Save the script so we can rerun it on every model\nversion change."},"start":228.544,"dur":5.469,"words":[{"text":"Try","s":228.844,"e":229.164},{"text":"it.","s":229.284,"e":229.604},{"text":"Have","s":229.804,"e":229.964},{"text":"your","s":230.044,"e":230.124},{"text":"agent","s":230.284,"e":230.444},{"text":"build","s":230.524,"e":230.684},{"text":"a","s":230.724,"e":230.764},{"text":"calibration","s":230.844,"e":231.404},{"text":"report","s":231.484,"e":231.884},{"text":"for","s":231.964,"e":232.044},{"text":"one","s":232.124,"e":232.204},{"text":"of","s":232.244,"e":232.284},{"text":"your","s":232.364,"e":232.444},{"text":"decisions.","s":232.524,"e":233.404}]},{"template":"recap","chapter":"Recap","narration":"So: calibrated means point eight is right eight times in ten. See it in a reliability diagram, summarize it with ECE and Brier. Chat training drifts toward overconfidence, but every model has to be measured on every task. And a simple recalibration fixes the numbers. Next: turning probabilities into actions with thresholds.","params":{"items":["Calibrated: of answers given 0.8, ~80% are right.","Reliability diagram + ECE + Brier, on your own labels.","It's per model, per task: measure, never assume.","Platt scaling fixes the numbers, not the ranking."],"next":"Confidence and thresholds","next_at":"Next"},"start":234.013,"dur":25.462,"words":[{"text":"So:","s":234.313,"e":235.033},{"text":"calibrated","s":235.153,"e":235.753},{"text":"means","s":235.833,"e":236.153},{"text":"point","s":236.473,"e":236.713},{"text":"eight","s":236.793,"e":236.953},{"text":"is","s":237.033,"e":237.193},{"text":"right","s":237.353,"e":237.513},{"text":"eight","s":237.593,"e":237.833},{"text":"times","s":238.033,"e":238.233},{"text":"in","s":238.313,"e":238.393},{"text":"ten.","s":238.473,"e":239.273},{"text":"See","s":239.353,"e":239.433},{"text":"it","s":239.513,"e":239.593},{"text":"in","s":239.673,"e":239.753},{"text":"a","s":239.793,"e":239.833},{"text":"reliability","s":239.913,"e":240.553},{"text":"diagram,","s":240.633,"e":241.433},{"text":"summarize","s":241.513,"e":241.993},{"text":"it","s":242.073,"e":242.153},{"text":"with","s":242.273,"e":242.393},{"text":"ECE","s":242.473,"e":243.193},{"text":"and","s":243.273,"e":243.353},{"text":"Brier.","s":243.433,"e":244.313},{"text":"Chat","s":244.433,"e":244.553},{"text":"training","s":244.633,"e":244.953},{"text":"drifts","s":245.033,"e":245.433},{"text":"toward","s":245.513,"e":245.753},{"text":"overconfidence,","s":245.833,"e":246.953},{"text":"but","s":247.113,"e":247.273},{"text":"every","s":247.353,"e":247.673},{"text":"model","s":247.753,"e":248.073},{"text":"has","s":248.153,"e":248.313},{"text":"to","s":248.393,"e":248.473},{"text":"be","s":248.553,"e":248.633},{"text":"measured","s":248.673,"e":248.953},{"text":"on","s":249.033,"e":249.273},{"text":"every","s":249.353,"e":249.673},{"text":"task.","s":249.913,"e":250.553},{"text":"And","s":250.593,"e":250.633},{"text":"a","s":250.713,"e":250.793},{"text":"simple","s":250.913,"e":251.033},{"text":"recalibration","s":251.113,"e":251.913},{"text":"fixes","s":251.993,"e":252.313},{"text":"the","s":252.353,"e":252.393},{"text":"numbers.","s":252.473,"e":253.593},{"text":"Next:","s":253.673,"e":254.473},{"text":"turning","s":254.593,"e":254.793},{"text":"probabilities","s":254.873,"e":255.673},{"text":"into","s":255.793,"e":255.913},{"text":"actions","s":255.993,"e":256.633},{"text":"with","s":256.713,"e":256.873},{"text":"thresholds.","s":256.953,"e":257.913}]}],"total":259.875,"audio":"/audio/09-calibration.mp3"}