{"id":"10-thresholds","number":10,"slug":"confidence-thresholds","title":"Confidence and thresholds","short":"Thresholds","module":"Probabilities you can act on","summary":"The answer tells you what; the probability tells you whether to act. Here is the three-path pattern (act, confirm, escalate), why thresholds should differ by action, a one-line formula that turns the cost of mistakes into a threshold, and the one thing thresholds cannot protect you from.","voice":"Jessica","scenes":[{"template":"title","chapter":"Intro","narration":"A probability on its own doesn't do anything. Your code turns it into behavior with thresholds. This lesson is about where to put them, and why.","params":{"kicker":"Lesson 10","module":"Module 3 · Probabilities you can act on","title":"Confidence and *thresholds*","subtitle":"The answer says what. The probability says whether to act.","cast":["curly","snap"]},"start":0.5,"dur":10.917,"words":[{"text":"A","s":0.8,"e":0.88},{"text":"probability","s":0.96,"e":1.6},{"text":"on","s":1.68,"e":1.76},{"text":"its","s":1.88,"e":2},{"text":"own","s":2.08,"e":2.48},{"text":"doesn't","s":2.64,"e":2.96},{"text":"do","s":3.04,"e":3.2},{"text":"anything.","s":3.48,"e":4},{"text":"Your","s":4.24,"e":4.4},{"text":"code","s":4.48,"e":4.8},{"text":"turns","s":4.92,"e":5.08},{"text":"it","s":5.12,"e":5.16},{"text":"into","s":5.2,"e":5.36},{"text":"behavior","s":5.44,"e":6.08},{"text":"with","s":6.16,"e":6.32},{"text":"thresholds.","s":6.4,"e":7.12},{"text":"This","s":7.6,"e":7.76},{"text":"lesson","s":7.96,"e":8.16},{"text":"is","s":8.24,"e":8.32},{"text":"about","s":8.4,"e":8.64},{"text":"where","s":8.72,"e":8.88},{"text":"to","s":8.96,"e":9.04},{"text":"put","s":9.12,"e":9.2},{"text":"them,","s":9.28,"e":9.68},{"text":"and","s":9.76,"e":9.84},{"text":"why.","s":9.92,"e":10.56}]},{"template":"statement","chapter":"A second axis","narration":"Here's the idea in one line, from the TypeSafe docs. Use confidence as a second axis. The answer tells you what. Confidence tells you whether to act.","params":{"eyebrow":"Confidence-gated routing","text":"The answer tells you *what*. Confidence tells you *whether to act*.","art":"gate","tone":"agent","punch":"whether to act"},"start":11.417,"dur":11.712,"words":[{"text":"Here's","s":11.717,"e":12.037},{"text":"the","s":12.077,"e":12.117},{"text":"idea","s":12.197,"e":12.597},{"text":"in","s":12.717,"e":12.837},{"text":"one","s":12.917,"e":13.077},{"text":"line,","s":13.157,"e":13.637},{"text":"from","s":13.717,"e":13.797},{"text":"the","s":13.837,"e":13.877},{"text":"TypeSafe","s":13.957,"e":14.357},{"text":"docs.","s":14.437,"e":15.117},{"text":"Use","s":15.397,"e":15.557},{"text":"confidence","s":15.637,"e":16.277},{"text":"as","s":16.357,"e":16.437},{"text":"a","s":16.517,"e":16.597},{"text":"second","s":16.677,"e":16.997},{"text":"axis.","s":17.157,"e":17.917},{"text":"The","s":18.117,"e":18.197},{"text":"answer","s":18.277,"e":18.677},{"text":"tells","s":18.757,"e":18.997},{"text":"you","s":19.077,"e":19.157},{"text":"what.","s":19.237,"e":19.637},{"text":"Confidence","s":20.037,"e":20.757},{"text":"tells","s":20.837,"e":21.077},{"text":"you","s":21.157,"e":21.237},{"text":"whether","s":21.317,"e":21.557},{"text":"to","s":21.637,"e":21.717},{"text":"act.","s":21.797,"e":22.517}]},{"template":"lanes","chapter":"Three paths","narration":"That gives you three paths. When the model is sure, act automatically. When it's fairly sure, ask the user to confirm, or gather more information. And when it isn't, hand the case to a person, or to a slower, smarter model. Watch where each request goes.","params":{"thresholds":[0.85,0.6],"lanes":[{"label":"Act automatically","rule":"≥ 0.85","tone":"good","icon":"bolt","at":"act automatically"},{"label":"Ask to confirm","rule":"0.60 – 0.85","tone":"warn","icon":"question","at":"confirm"},{"label":"Escalate","rule":"< 0.60 · person or slower model","tone":"human","who":"you","at":"hand the case"}],"items":[{"text":"“What's my balance?”","conf":0.98},{"text":"“Approve the transfer”","conf":0.91},{"text":"“Approve it, I think?”","conf":0.72},{"text":"“Hmm, the thing from yesterday”","conf":0.41},{"text":"“Move the money, the usual”","conf":0.66}]},"start":23.129,"dur":18.348,"words":[{"text":"That","s":23.429,"e":23.669},{"text":"gives","s":23.749,"e":23.909},{"text":"you","s":24.029,"e":24.149},{"text":"three","s":24.309,"e":24.469},{"text":"paths.","s":24.749,"e":25.429},{"text":"When","s":25.509,"e":25.589},{"text":"the","s":25.629,"e":25.669},{"text":"model","s":25.829,"e":25.989},{"text":"is","s":26.069,"e":26.149},{"text":"sure,","s":26.349,"e":26.869},{"text":"act","s":27.029,"e":27.189},{"text":"automatically.","s":27.269,"e":28.389},{"text":"When","s":28.469,"e":28.549},{"text":"it's","s":28.589,"e":28.789},{"text":"fairly","s":28.909,"e":29.189},{"text":"sure,","s":29.269,"e":29.909},{"text":"ask","s":30.069,"e":30.229},{"text":"the","s":30.269,"e":30.309},{"text":"user","s":30.389,"e":30.629},{"text":"to","s":30.669,"e":30.709},{"text":"confirm,","s":30.789,"e":31.509},{"text":"or","s":31.629,"e":31.749},{"text":"gather","s":31.829,"e":32.149},{"text":"more","s":32.229,"e":32.309},{"text":"information.","s":32.669,"e":33.429},{"text":"And","s":33.509,"e":33.589},{"text":"when","s":33.629,"e":33.669},{"text":"it","s":33.749,"e":33.909},{"text":"isn't,","s":33.989,"e":34.629},{"text":"hand","s":34.709,"e":34.869},{"text":"the","s":34.909,"e":34.949},{"text":"case","s":35.029,"e":35.269},{"text":"to","s":35.309,"e":35.349},{"text":"a","s":35.429,"e":35.509},{"text":"person,","s":35.749,"e":36.229},{"text":"or","s":36.349,"e":36.469},{"text":"to","s":36.549,"e":36.629},{"text":"a","s":36.709,"e":36.789},{"text":"slower,","s":36.869,"e":37.429},{"text":"smarter","s":37.509,"e":37.829},{"text":"model.","s":37.909,"e":38.709},{"text":"Watch","s":38.869,"e":39.029},{"text":"where","s":39.149,"e":39.269},{"text":"each","s":39.349,"e":39.509},{"text":"request","s":39.749,"e":39.989},{"text":"goes.","s":40.069,"e":40.869}]},{"template":"code","chapter":"Thresholds per action","narration":"But one threshold for everything is a mistake. Thresholds belong to actions. In this voice banking example, checking a balance is harmless if it's wrong, so point six is enough. Approving a transfer is not, so it needs point eight five to go through, and anything less gets a spoken confirmation. And below point six, nothing happens without a person.","params":{"file":"voice_banking.py","lang":"py","lines":["intent = answers[\"intent\"]","if intent.confidence < 0.6:","    route_to_support_agent(account)        # genuinely unsure","elif intent.choice == \"check_balance\":","    show_balance(account)                  # low stakes: 0.6 is fine","elif intent.choice == \"approve_transfer\":","    if intent.confidence > 0.85:","        approve_transfer(account)          # high stakes, high confidence","    else:","        ask_user_to_confirm(account)       # high stakes, moderate"],"hl":[{"from":4,"to":5,"label":"cheap mistake","tone":"good","at":"harmless"},{"from":7,"to":10,"label":"expensive mistake","tone":"bad","at":"Approving a transfer"}]},"start":41.477,"dur":24.199,"words":[{"text":"But","s":41.777,"e":42.017},{"text":"one","s":42.097,"e":42.257},{"text":"threshold","s":42.337,"e":42.817},{"text":"for","s":42.937,"e":43.057},{"text":"everything","s":43.217,"e":43.857},{"text":"is","s":43.937,"e":44.017},{"text":"a","s":44.057,"e":44.097},{"text":"mistake.","s":44.257,"e":45.137},{"text":"Thresholds","s":45.217,"e":45.777},{"text":"belong","s":45.857,"e":46.097},{"text":"to","s":46.177,"e":46.337},{"text":"actions.","s":46.697,"e":47.457},{"text":"In","s":47.537,"e":47.617},{"text":"this","s":47.697,"e":47.777},{"text":"voice","s":47.857,"e":48.017},{"text":"banking","s":48.097,"e":48.417},{"text":"example,","s":48.497,"e":49.217},{"text":"checking","s":49.377,"e":49.537},{"text":"a","s":49.617,"e":49.697},{"text":"balance","s":49.897,"e":50.097},{"text":"is","s":50.177,"e":50.337},{"text":"harmless","s":50.417,"e":50.817},{"text":"if","s":50.857,"e":50.897},{"text":"it's","s":50.977,"e":51.137},{"text":"wrong,","s":51.297,"e":51.697},{"text":"so","s":51.857,"e":52.017},{"text":"point","s":52.097,"e":52.337},{"text":"six","s":52.417,"e":52.657},{"text":"is","s":52.737,"e":52.817},{"text":"enough.","s":52.897,"e":53.697},{"text":"Approving","s":53.817,"e":54.097},{"text":"a","s":54.177,"e":54.257},{"text":"transfer","s":54.577,"e":54.897},{"text":"is","s":54.977,"e":55.217},{"text":"not,","s":55.417,"e":55.777},{"text":"so","s":55.857,"e":56.097},{"text":"it","s":56.177,"e":56.257},{"text":"needs","s":56.337,"e":56.577},{"text":"point","s":56.737,"e":56.897},{"text":"eight","s":57.017,"e":57.137},{"text":"five","s":57.217,"e":57.457},{"text":"to","s":57.537,"e":57.617},{"text":"go","s":57.657,"e":57.697},{"text":"through,","s":57.857,"e":58.537},{"text":"and","s":58.577,"e":58.817},{"text":"anything","s":59.017,"e":59.217},{"text":"less","s":59.377,"e":59.537},{"text":"gets","s":59.617,"e":59.777},{"text":"a","s":59.817,"e":59.857},{"text":"spoken","s":60.057,"e":60.257},{"text":"confirmation.","s":60.337,"e":61.537},{"text":"And","s":61.617,"e":61.697},{"text":"below","s":61.857,"e":62.017},{"text":"point","s":62.097,"e":62.337},{"text":"six,","s":62.417,"e":63.057},{"text":"nothing","s":63.137,"e":63.457},{"text":"happens","s":63.537,"e":63.857},{"text":"without","s":63.937,"e":64.177},{"text":"a","s":64.217,"e":64.257},{"text":"person.","s":64.497,"e":64.977}]},{"template":"formula","chapter":"A threshold from costs","narration":"Where should a threshold sit? There's a formula, from decision theory. If acting on a false yes costs C false, and missing a true yes costs C miss, act when the probability is above C false divided by C false plus C miss. It assumes the probability is calibrated, which is why lesson nine came first.","params":{"title":"Act when p is above:","parts":[{"text":"C_false","sub":"cost of acting when it's actually no","tone":"bad","at":"acting on a false yes"},{"op":"÷"},{"text":"C_false + C_miss","sub":"C_miss: cost of missing a real yes","tone":"paper","at":"missing a true yes"}],"caption":"Charles Elkan, The Foundations of Cost-Sensitive Learning (2001). Requires calibrated probabilities.","caption_at":"decision theory"},"start":65.676,"dur":22.109,"words":[{"text":"Where","s":65.976,"e":66.216},{"text":"should","s":66.296,"e":66.376},{"text":"a","s":66.456,"e":66.536},{"text":"threshold","s":66.616,"e":67.176},{"text":"sit?","s":67.336,"e":67.816},{"text":"There's","s":68.216,"e":68.376},{"text":"a","s":68.456,"e":68.496},{"text":"formula,","s":68.536,"e":69.176},{"text":"from","s":69.256,"e":69.336},{"text":"decision","s":69.416,"e":69.816},{"text":"theory.","s":69.896,"e":70.456},{"text":"If","s":70.696,"e":70.776},{"text":"acting","s":70.856,"e":71.176},{"text":"on","s":71.216,"e":71.256},{"text":"a","s":71.336,"e":71.416},{"text":"false","s":71.496,"e":71.736},{"text":"yes","s":71.896,"e":72.216},{"text":"costs","s":72.296,"e":72.696},{"text":"C","s":72.856,"e":73.096},{"text":"false,","s":73.176,"e":73.816},{"text":"and","s":73.896,"e":73.976},{"text":"missing","s":74.056,"e":74.296},{"text":"a","s":74.336,"e":74.376},{"text":"true","s":74.456,"e":74.696},{"text":"yes","s":74.776,"e":75.096},{"text":"costs","s":75.176,"e":75.576},{"text":"C","s":75.656,"e":75.896},{"text":"miss,","s":75.976,"e":76.856},{"text":"act","s":77.016,"e":77.176},{"text":"when","s":77.216,"e":77.256},{"text":"the","s":77.336,"e":77.416},{"text":"probability","s":77.496,"e":78.136},{"text":"is","s":78.216,"e":78.296},{"text":"above","s":78.496,"e":78.696},{"text":"C","s":78.856,"e":79.016},{"text":"false","s":79.096,"e":79.496},{"text":"divided","s":79.576,"e":79.976},{"text":"by","s":80.056,"e":80.296},{"text":"C","s":80.456,"e":80.696},{"text":"false","s":80.896,"e":81.096},{"text":"plus","s":81.216,"e":81.336},{"text":"C","s":81.416,"e":81.576},{"text":"miss.","s":81.656,"e":82.136},{"text":"It","s":82.616,"e":82.696},{"text":"assumes","s":82.776,"e":83.096},{"text":"the","s":83.176,"e":83.216},{"text":"probability","s":83.256,"e":83.816},{"text":"is","s":83.896,"e":83.976},{"text":"calibrated,","s":84.056,"e":84.936},{"text":"which","s":85.016,"e":85.096},{"text":"is","s":85.136,"e":85.176},{"text":"why","s":85.256,"e":85.336},{"text":"lesson","s":85.416,"e":85.656},{"text":"nine","s":85.816,"e":86.056},{"text":"came","s":86.136,"e":86.376},{"text":"first.","s":86.616,"e":87.176}]},{"template":"stat","chapter":"A threshold from costs","narration":"Two examples. Auto-approving a refund that wasn't owed costs you forty dollars, and making a real refund wait for review costs about ten. Forty over fifty: act above point eight. Now a safety report. A false alarm costs a minute of someone's time, a missed one could cost a hundred times more. One over a hundred and one: flag anything above about point oh one.","params":{"items":[{"value":0.8,"decimals":2,"label":"auto-refund: $40 / ($40 + $10)","tone":"agent","at":"point eight"},{"value":0.01,"decimals":2,"label":"safety flag: 1 / (1 + 100)","tone":"bad","at":"point oh one","hero":true}],"src":"Illustrative costs. Put your own in, and write them next to the constant."},"start":87.785,"dur":27.699,"words":[{"text":"Two","s":88.085,"e":88.245},{"text":"examples.","s":88.325,"e":89.205},{"text":"Auto-approving","s":89.445,"e":90.165},{"text":"a","s":90.218,"e":90.272},{"text":"refund","s":90.325,"e":90.725},{"text":"that","s":90.805,"e":90.885},{"text":"wasn't","s":91.045,"e":91.365},{"text":"owed","s":91.445,"e":91.765},{"text":"costs","s":91.845,"e":92.165},{"text":"you","s":92.285,"e":92.405},{"text":"forty","s":92.525,"e":92.725},{"text":"dollars,","s":92.805,"e":93.685},{"text":"and","s":93.845,"e":94.005},{"text":"making","s":94.085,"e":94.325},{"text":"a","s":94.405,"e":94.485},{"text":"real","s":94.565,"e":94.725},{"text":"refund","s":94.805,"e":95.285},{"text":"wait","s":95.405,"e":95.525},{"text":"for","s":95.565,"e":95.605},{"text":"review","s":95.685,"e":96.165},{"text":"costs","s":96.325,"e":96.605},{"text":"about","s":96.645,"e":96.965},{"text":"ten.","s":97.165,"e":98.005},{"text":"Forty","s":98.165,"e":98.445},{"text":"over","s":98.565,"e":98.805},{"text":"fifty:","s":98.885,"e":99.685},{"text":"act","s":99.845,"e":100.005},{"text":"above","s":100.085,"e":100.485},{"text":"point","s":100.565,"e":100.885},{"text":"eight.","s":101.045,"e":101.765},{"text":"Now","s":101.845,"e":102.005},{"text":"a","s":102.165,"e":102.205},{"text":"safety","s":102.245,"e":102.565},{"text":"report.","s":102.645,"e":103.445},{"text":"A","s":103.525,"e":103.605},{"text":"false","s":103.725,"e":103.845},{"text":"alarm","s":103.925,"e":104.325},{"text":"costs","s":104.485,"e":104.725},{"text":"a","s":104.805,"e":104.885},{"text":"minute","s":104.965,"e":105.205},{"text":"of","s":105.285,"e":105.365},{"text":"someone's","s":105.525,"e":105.765},{"text":"time,","s":105.965,"e":106.565},{"text":"a","s":106.605,"e":106.645},{"text":"missed","s":106.725,"e":107.045},{"text":"one","s":107.125,"e":107.365},{"text":"could","s":107.445,"e":107.525},{"text":"cost","s":107.605,"e":107.845},{"text":"a","s":107.925,"e":108.085},{"text":"hundred","s":108.165,"e":108.485},{"text":"times","s":108.645,"e":108.805},{"text":"more.","s":108.885,"e":109.845},{"text":"One","s":109.925,"e":110.165},{"text":"over","s":110.245,"e":110.485},{"text":"a","s":110.565,"e":110.645},{"text":"hundred","s":110.685,"e":110.965},{"text":"and","s":111.045,"e":111.125},{"text":"one:","s":111.205,"e":111.845},{"text":"flag","s":112.045,"e":112.245},{"text":"anything","s":112.405,"e":112.885},{"text":"above","s":112.965,"e":113.285},{"text":"about","s":113.485,"e":113.685},{"text":"point","s":113.805,"e":113.925},{"text":"oh","s":114.085,"e":114.245},{"text":"one.","s":114.405,"e":114.885}]},{"template":"quote","chapter":"A threshold from costs","narration":"Developer Daniel Sam put the attitude well. Fifty percent is a coin toss you ignore. Ninety five percent is a bet you take. Thresholds should scale with the cost of being wrong.","params":{"text":"50% is a coin toss you ignore, 95% is a bet you take. *Thresholds should scale with the cost of being wrong.*","who":"Daniel Sam","role":"developer","src":"The Model That Doesn't Talk"},"start":115.484,"dur":13.071,"words":[{"text":"Developer","s":115.784,"e":116.264},{"text":"Daniel","s":116.424,"e":116.744},{"text":"Sam","s":116.824,"e":117.224},{"text":"put","s":117.304,"e":117.384},{"text":"the","s":117.424,"e":117.464},{"text":"attitude","s":117.544,"e":117.944},{"text":"well.","s":118.024,"e":118.624},{"text":"Fifty","s":118.904,"e":119.144},{"text":"percent","s":119.224,"e":119.704},{"text":"is","s":119.784,"e":119.864},{"text":"a","s":119.944,"e":120.024},{"text":"coin","s":120.144,"e":120.264},{"text":"toss","s":120.344,"e":120.664},{"text":"you","s":120.744,"e":120.824},{"text":"ignore.","s":120.904,"e":121.864},{"text":"Ninety","s":121.904,"e":122.104},{"text":"five","s":122.264,"e":122.424},{"text":"percent","s":122.584,"e":122.984},{"text":"is","s":123.144,"e":123.224},{"text":"a","s":123.304,"e":123.344},{"text":"bet","s":123.384,"e":123.544},{"text":"you","s":123.624,"e":123.784},{"text":"take.","s":123.984,"e":124.664},{"text":"Thresholds","s":124.744,"e":125.304},{"text":"should","s":125.384,"e":125.624},{"text":"scale","s":125.784,"e":126.104},{"text":"with","s":126.184,"e":126.264},{"text":"the","s":126.344,"e":126.424},{"text":"cost","s":126.504,"e":126.744},{"text":"of","s":126.824,"e":126.904},{"text":"being","s":127.024,"e":127.144},{"text":"wrong.","s":127.224,"e":127.944}]},{"template":"statement","chapter":"Who owns the threshold","narration":"Some see this as a catch. Armin Ronacher put it as delegating the hallucination problem a little bit to the user. He's right, and that's the deal. The threshold is a business decision, and now it's yours: written in code, readable, and changeable in one line.","params":{"eyebrow":"Armin Ronacher, via TechCrunch","text":"It “delegates the hallucination problem a little bit to the user.” *Yes. That's the deal.*","sub":"Thresholds are business decisions. Now they live in code you can read and change.","sub_at":"business decision","art":"curly","punch":"that's the deal"},"start":128.555,"dur":18.661,"words":[{"text":"Some","s":128.855,"e":129.065},{"text":"see","s":129.095,"e":129.282},{"text":"this","s":129.335,"e":129.545},{"text":"as","s":129.575,"e":129.695},{"text":"a","s":129.735,"e":129.788},{"text":"catch.","s":129.842,"e":130.375},{"text":"Armin","s":130.455,"e":131.095},{"text":"Ronacher","s":131.175,"e":131.655},{"text":"put","s":131.735,"e":131.815},{"text":"it","s":131.895,"e":131.975},{"text":"as","s":132.055,"e":132.295},{"text":"delegating","s":132.535,"e":133.095},{"text":"the","s":133.135,"e":133.175},{"text":"hallucination","s":133.255,"e":133.975},{"text":"problem","s":134.055,"e":134.535},{"text":"a","s":134.615,"e":134.695},{"text":"little","s":134.855,"e":135.015},{"text":"bit","s":135.095,"e":135.255},{"text":"to","s":135.295,"e":135.335},{"text":"the","s":135.415,"e":135.495},{"text":"user.","s":135.575,"e":136.055},{"text":"He's","s":136.455,"e":136.615},{"text":"right,","s":136.695,"e":137.255},{"text":"and","s":137.335,"e":137.415},{"text":"that's","s":137.495,"e":137.655},{"text":"the","s":137.695,"e":137.735},{"text":"deal.","s":137.815,"e":138.335},{"text":"The","s":138.535,"e":138.615},{"text":"threshold","s":138.695,"e":139.175},{"text":"is","s":139.255,"e":139.335},{"text":"a","s":139.415,"e":139.495},{"text":"business","s":139.575,"e":139.975},{"text":"decision,","s":140.255,"e":140.775},{"text":"and","s":140.855,"e":140.935},{"text":"now","s":141.015,"e":141.255},{"text":"it's","s":141.495,"e":141.655},{"text":"yours:","s":141.695,"e":142.535},{"text":"written","s":142.655,"e":142.775},{"text":"in","s":142.855,"e":142.935},{"text":"code,","s":143.015,"e":143.655},{"text":"readable,","s":143.735,"e":144.375},{"text":"and","s":144.455,"e":144.535},{"text":"changeable","s":144.615,"e":145.255},{"text":"in","s":145.335,"e":145.495},{"text":"one","s":145.655,"e":145.815},{"text":"line.","s":145.895,"e":146.615}]},{"template":"statement","chapter":"What thresholds can't catch","narration":"One thing thresholds can't do. The author of Laya, an open decision model, fed its English model text in other languages. Accuracy collapsed, all the way to zero on some, but its average confidence never dropped below point eight eight five. A model can be confidently lost. Check that inputs are in scope before you trust the number.","params":{"eyebrow":"Laya, on out-of-domain inputs","text":"Confidence stayed above *0.885* even where accuracy was *0%*.","sub":"Out-of-scope inputs (another language, an unknown domain) need a check before the decision.","sub_at":"Check that inputs are in scope","art":"warn","tone":"bad","punch":"confidently lost"},"start":147.216,"dur":23.232,"words":[{"text":"One","s":147.516,"e":147.756},{"text":"thing","s":147.836,"e":148.076},{"text":"thresholds","s":148.156,"e":148.716},{"text":"can't","s":148.876,"e":149.196},{"text":"do.","s":149.276,"e":149.596},{"text":"The","s":150.076,"e":150.156},{"text":"author","s":150.236,"e":150.476},{"text":"of","s":150.556,"e":150.636},{"text":"Laya,","s":150.716,"e":151.276},{"text":"an","s":151.356,"e":151.436},{"text":"open","s":151.516,"e":151.756},{"text":"decision","s":151.836,"e":152.156},{"text":"model,","s":152.236,"e":152.876},{"text":"fed","s":152.956,"e":153.116},{"text":"its","s":153.196,"e":153.356},{"text":"English","s":153.516,"e":153.676},{"text":"model","s":153.756,"e":153.996},{"text":"text","s":154.076,"e":154.396},{"text":"in","s":154.476,"e":154.556},{"text":"other","s":154.636,"e":154.796},{"text":"languages.","s":154.876,"e":155.596},{"text":"Accuracy","s":155.996,"e":156.556},{"text":"collapsed,","s":156.716,"e":157.676},{"text":"all","s":157.756,"e":157.836},{"text":"the","s":157.876,"e":157.916},{"text":"way","s":157.996,"e":158.076},{"text":"to","s":158.116,"e":158.156},{"text":"zero","s":158.316,"e":158.636},{"text":"on","s":158.716,"e":158.876},{"text":"some,","s":158.956,"e":159.756},{"text":"but","s":159.796,"e":159.836},{"text":"its","s":159.916,"e":160.076},{"text":"average","s":160.276,"e":160.476},{"text":"confidence","s":160.556,"e":161.116},{"text":"never","s":161.356,"e":161.596},{"text":"dropped","s":161.676,"e":161.996},{"text":"below","s":162.076,"e":162.236},{"text":"point","s":162.396,"e":162.556},{"text":"eight","s":162.716,"e":162.796},{"text":"eight","s":162.956,"e":163.116},{"text":"five.","s":163.316,"e":163.996},{"text":"A","s":164.036,"e":164.076},{"text":"model","s":164.156,"e":164.476},{"text":"can","s":164.556,"e":164.716},{"text":"be","s":164.796,"e":164.876},{"text":"confidently","s":164.956,"e":165.596},{"text":"lost.","s":165.676,"e":166.396},{"text":"Check","s":166.636,"e":166.796},{"text":"that","s":166.876,"e":166.956},{"text":"inputs","s":167.036,"e":167.356},{"text":"are","s":167.396,"e":167.436},{"text":"in","s":167.516,"e":167.596},{"text":"scope","s":167.676,"e":168.076},{"text":"before","s":168.156,"e":168.556},{"text":"you","s":168.636,"e":168.716},{"text":"trust","s":168.836,"e":168.956},{"text":"the","s":169.036,"e":169.116},{"text":"number.","s":169.316,"e":169.836}]},{"template":"quiz","chapter":"Quick check","narration":"Quick check. Wrongly auto-closing a ticket costs nine times more than sending a closable ticket to review. With calibrated probabilities, where should the auto-close threshold be?","params":{"q":"A wrong auto-close costs **9×** a needless review. Threshold?","options":["0.5","0.75","0.9","0.99"],"answer":2,"explain":"C_false ÷ (C_false + C_miss) = 9 ÷ (9 + 1) = 0.9. Auto-close above 0.9; review the rest."},"start":170.448,"dur":16.498,"words":[{"text":"Quick","s":170.748,"e":170.964},{"text":"check.","s":170.988,"e":171.388},{"text":"Wrongly","s":171.588,"e":172.188},{"text":"auto-closing","s":172.268,"e":172.828},{"text":"a","s":172.908,"e":172.988},{"text":"ticket","s":173.068,"e":173.388},{"text":"costs","s":173.468,"e":173.948},{"text":"nine","s":174.108,"e":174.348},{"text":"times","s":174.428,"e":174.748},{"text":"more","s":174.908,"e":175.148},{"text":"than","s":175.228,"e":175.308},{"text":"sending","s":175.388,"e":175.628},{"text":"a","s":175.708,"e":175.788},{"text":"closable","s":175.868,"e":176.268},{"text":"ticket","s":176.348,"e":176.588},{"text":"to","s":176.668,"e":176.748},{"text":"review.","s":176.988,"e":177.308},{"text":"With","s":177.708,"e":177.868},{"text":"calibrated","s":177.948,"e":178.428},{"text":"probabilities,","s":178.508,"e":179.548},{"text":"where","s":179.628,"e":179.868},{"text":"should","s":179.948,"e":180.028},{"text":"the","s":180.108,"e":180.188},{"text":"auto-close","s":180.268,"e":180.748},{"text":"threshold","s":180.828,"e":181.228},{"text":"be?","s":181.308,"e":181.788}]},{"template":"exercise","chapter":"Try it","narration":"Try it. Have your agent write the threshold policy for one decision, with the cost reasoning in the code.","params":{"title":"Write a threshold policy","intro":"Paste this into your coding agent:","prompt":"For one automated decision in this codebase, write a threshold policy:\n- list each action it can trigger and estimate the cost of a wrong\n  action vs a missed one (ask me if you can't infer them)\n- compute each threshold as C_false / (C_false + C_miss)\n- add a review band that goes to a person or a slower model\n- add a check that rejects out-of-scope inputs before the decision\nPut thresholds in named constants with the cost reasoning in comments,\nand log probability, threshold and action for every decision."},"start":186.946,"dur":7.219,"words":[{"text":"Try","s":187.246,"e":187.486},{"text":"it.","s":187.566,"e":187.886},{"text":"Have","s":188.206,"e":188.366},{"text":"your","s":188.446,"e":188.526},{"text":"agent","s":188.606,"e":188.846},{"text":"write","s":188.926,"e":189.086},{"text":"the","s":189.126,"e":189.166},{"text":"threshold","s":189.246,"e":189.646},{"text":"policy","s":189.726,"e":190.206},{"text":"for","s":190.286,"e":190.446},{"text":"one","s":190.526,"e":190.686},{"text":"decision,","s":190.766,"e":191.566},{"text":"with","s":191.646,"e":191.726},{"text":"the","s":191.766,"e":191.806},{"text":"cost","s":191.886,"e":192.206},{"text":"reasoning","s":192.366,"e":192.686},{"text":"in","s":192.726,"e":192.766},{"text":"the","s":192.846,"e":192.886},{"text":"code.","s":192.926,"e":193.566}]},{"template":"recap","chapter":"Recap","narration":"So: the answer says what, the probability says whether. Act, confirm, or escalate. Give every action its own threshold, set by what mistakes cost. And check that inputs are in scope, because a model can be confidently lost. Next: how much can you automate, and how to pick thresholds from data.","params":{"items":["The answer says what; the probability says whether.","Three paths: act, confirm, escalate.","Threshold per action: C_false ÷ (C_false + C_miss).","Confidence can't catch out-of-scope inputs: check them first."],"next":"Automate the sure ones","next_at":"Next"},"start":194.165,"dur":23.059,"words":[{"text":"So:","s":194.465,"e":195.105},{"text":"the","s":195.185,"e":195.265},{"text":"answer","s":195.345,"e":195.585},{"text":"says","s":195.665,"e":195.905},{"text":"what,","s":195.985,"e":196.625},{"text":"the","s":196.665,"e":196.705},{"text":"probability","s":196.785,"e":197.505},{"text":"says","s":197.665,"e":197.905},{"text":"whether.","s":198.065,"e":198.705},{"text":"Act,","s":198.945,"e":199.425},{"text":"confirm,","s":199.505,"e":200.225},{"text":"or","s":200.265,"e":200.305},{"text":"escalate.","s":200.465,"e":201.305},{"text":"Give","s":201.505,"e":201.745},{"text":"every","s":201.905,"e":202.065},{"text":"action","s":202.145,"e":202.465},{"text":"its","s":202.545,"e":202.785},{"text":"own","s":202.865,"e":203.025},{"text":"threshold,","s":203.105,"e":203.905},{"text":"set","s":203.985,"e":204.225},{"text":"by","s":204.305,"e":204.385},{"text":"what","s":204.465,"e":204.545},{"text":"mistakes","s":204.625,"e":205.105},{"text":"cost.","s":205.185,"e":206.065},{"text":"And","s":206.145,"e":206.225},{"text":"check","s":206.345,"e":206.465},{"text":"that","s":206.545,"e":206.625},{"text":"inputs","s":206.705,"e":207.025},{"text":"are","s":207.105,"e":207.185},{"text":"in","s":207.225,"e":207.265},{"text":"scope,","s":207.425,"e":207.985},{"text":"because","s":208.105,"e":208.225},{"text":"a","s":208.305,"e":208.385},{"text":"model","s":208.585,"e":208.785},{"text":"can","s":208.865,"e":208.945},{"text":"be","s":209.025,"e":209.185},{"text":"confidently","s":209.345,"e":209.905},{"text":"lost.","s":210.145,"e":210.945},{"text":"Next:","s":211.025,"e":211.825},{"text":"how","s":211.905,"e":211.985},{"text":"much","s":212.065,"e":212.225},{"text":"can","s":212.305,"e":212.385},{"text":"you","s":212.465,"e":212.545},{"text":"automate,","s":212.705,"e":213.345},{"text":"and","s":213.465,"e":213.585},{"text":"how","s":213.745,"e":213.825},{"text":"to","s":213.905,"e":213.985},{"text":"pick","s":214.065,"e":214.145},{"text":"thresholds","s":214.225,"e":214.785},{"text":"from","s":214.865,"e":214.945},{"text":"data.","s":215.025,"e":215.665}]}],"total":217.624,"audio":"/audio/10-thresholds.mp3"}