{"id":"11-automate-the-sure-ones","number":11,"slug":"selective-automation","title":"Automate the sure ones","short":"Automate the sure ones","module":"Probabilities you can act on","summary":"You don't need a model that is right about everything. You need to know which answers you can trust. Here is the coverage curve that answers 'how much can we automate at 95%?', how to pick thresholds from a few hundred labels, what confidence-ordered review saves on our own data, and how to answer less precisely instead of wrongly.","voice":"Jessica","scenes":[{"template":"title","chapter":"Intro","narration":"No model is right about everything. But you don't need that. You need to know which of its answers you can trust, and send the rest somewhere else. That's selective automation.","params":{"kicker":"Lesson 11","module":"Module 3 · Probabilities you can act on","title":"Automate *the sure ones*","subtitle":"Know which answers to trust, and route the rest.","cast":["snap","you"]},"start":0.5,"dur":12.198,"words":[{"text":"No","s":0.8,"e":1.04},{"text":"model","s":1.12,"e":1.44},{"text":"is","s":1.52,"e":1.68},{"text":"right","s":1.76,"e":1.92},{"text":"about","s":2.04,"e":2.16},{"text":"everything.","s":2.32,"e":2.96},{"text":"But","s":3.36,"e":3.6},{"text":"you","s":3.68,"e":3.76},{"text":"don't","s":3.8,"e":3.92},{"text":"need","s":4,"e":4.16},{"text":"that.","s":4.24,"e":4.56},{"text":"You","s":4.96,"e":5.04},{"text":"need","s":5.12,"e":5.2},{"text":"to","s":5.28,"e":5.36},{"text":"know","s":5.44,"e":5.52},{"text":"which","s":5.68,"e":5.84},{"text":"of","s":5.92,"e":5.96},{"text":"its","s":6,"e":6.16},{"text":"answers","s":6.36,"e":6.56},{"text":"you","s":6.6,"e":6.64},{"text":"can","s":6.72,"e":6.8},{"text":"trust,","s":6.88,"e":7.52},{"text":"and","s":7.64,"e":7.76},{"text":"send","s":7.84,"e":8},{"text":"the","s":8.08,"e":8.12},{"text":"rest","s":8.16,"e":8.4},{"text":"somewhere","s":8.56,"e":8.8},{"text":"else.","s":8.88,"e":9.48},{"text":"That's","s":9.76,"e":10.08},{"text":"selective","s":10.24,"e":10.8},{"text":"automation.","s":10.88,"e":11.84}]},{"template":"quote","chapter":"The real question","narration":"On launch day, a Hacker News commenter asked the question that matters. Set a confidence threshold, trust the decisions above it, reject the rest. Then ask: what percent of decisions can we automate, and still hit ninety percent accuracy?","params":{"text":"What % of decisions can we automate *to achieve 90% accuracy*?","who":"ActivePattern","role":"Hacker News, Jev launch thread","src":"news.ycombinator.com"},"start":12.698,"dur":15.944,"words":[{"text":"On","s":12.998,"e":13.158},{"text":"launch","s":13.318,"e":13.478},{"text":"day,","s":13.558,"e":14.038},{"text":"a","s":14.078,"e":14.118},{"text":"Hacker","s":14.158,"e":14.438},{"text":"News","s":14.518,"e":14.678},{"text":"commenter","s":14.758,"e":15.238},{"text":"asked","s":15.318,"e":15.558},{"text":"the","s":15.598,"e":15.638},{"text":"question","s":15.718,"e":16.118},{"text":"that","s":16.198,"e":16.358},{"text":"matters.","s":16.438,"e":17.238},{"text":"Set","s":17.478,"e":17.638},{"text":"a","s":17.678,"e":17.718},{"text":"confidence","s":17.798,"e":18.198},{"text":"threshold,","s":18.278,"e":19.158},{"text":"trust","s":19.318,"e":19.478},{"text":"the","s":19.518,"e":19.558},{"text":"decisions","s":19.638,"e":20.038},{"text":"above","s":20.158,"e":20.278},{"text":"it,","s":20.358,"e":20.758},{"text":"reject","s":20.838,"e":21.318},{"text":"the","s":21.398,"e":21.478},{"text":"rest.","s":21.558,"e":22.198},{"text":"Then","s":22.438,"e":22.598},{"text":"ask:","s":22.678,"e":23.398},{"text":"what","s":23.478,"e":23.558},{"text":"percent","s":23.638,"e":23.878},{"text":"of","s":23.918,"e":23.958},{"text":"decisions","s":24.038,"e":24.518},{"text":"can","s":24.598,"e":24.678},{"text":"we","s":24.718,"e":24.758},{"text":"automate,","s":24.918,"e":25.438},{"text":"and","s":25.478,"e":25.638},{"text":"still","s":25.718,"e":26.038},{"text":"hit","s":26.118,"e":26.278},{"text":"ninety","s":26.438,"e":26.678},{"text":"percent","s":26.758,"e":27.078},{"text":"accuracy?","s":27.158,"e":28.038}]},{"template":"coverage","chapter":"The coverage curve","narration":"Here's that answer for Jev on our 3,080 banking messages. Automate everything, and it's eighty percent right. Only automate answers above point nine, and you handle sixty nine percent of messages at ninety two percent. Above point nine nine: half of all messages, at ninety six percent. To hit ninety five, you can automate about fifty seven percent.","params":{"title":"How much can we automate?","curve":[{"t":0,"coverage":1,"acc":0.801},{"t":0.5,"coverage":0.964,"acc":0.817},{"t":0.7,"coverage":0.842,"acc":0.862},{"t":0.8,"coverage":0.778,"acc":0.892},{"t":0.9,"coverage":0.687,"acc":0.92},{"t":0.95,"coverage":0.621,"acc":0.939},{"t":0.99,"coverage":0.503,"acc":0.963},{"t":1,"coverage":0.425,"acc":0.973}],"stops":[{"t":0,"coverage":1,"acc":0.801,"at":"Automate everything"},{"t":0.9,"coverage":0.687,"acc":0.92,"at":"above point nine"},{"t":0.99,"coverage":0.503,"acc":0.963,"at":"point nine nine","note":"≈57% of messages clear 95%"}],"target":{"acc":0.95,"label":"95% target","at":"To hit ninety five"},"ymin":0.78,"curve_at":"Here's that answer"},"start":28.642,"dur":24.121,"words":[{"text":"Here's","s":28.942,"e":29.262},{"text":"that","s":29.342,"e":29.422},{"text":"answer","s":29.582,"e":29.742},{"text":"for","s":29.822,"e":29.902},{"text":"Jev","s":29.982,"e":30.382},{"text":"on","s":30.462,"e":30.542},{"text":"our","s":30.662,"e":30.782},{"text":"3,080","s":30.862,"e":31.742},{"text":"banking","s":31.822,"e":32.142},{"text":"messages.","s":32.222,"e":33.022},{"text":"Automate","s":33.262,"e":33.662},{"text":"everything,","s":33.902,"e":34.782},{"text":"and","s":34.942,"e":35.102},{"text":"it's","s":35.182,"e":35.502},{"text":"eighty","s":35.582,"e":35.742},{"text":"percent","s":35.822,"e":36.062},{"text":"right.","s":36.142,"e":36.702},{"text":"Only","s":36.942,"e":37.182},{"text":"automate","s":37.262,"e":37.582},{"text":"answers","s":37.662,"e":37.982},{"text":"above","s":38.062,"e":38.382},{"text":"point","s":38.542,"e":38.702},{"text":"nine,","s":38.862,"e":39.262},{"text":"and","s":39.342,"e":39.422},{"text":"you","s":39.462,"e":39.502},{"text":"handle","s":39.582,"e":39.822},{"text":"sixty","s":39.982,"e":40.302},{"text":"nine","s":40.382,"e":40.462},{"text":"percent","s":40.542,"e":40.782},{"text":"of","s":40.862,"e":40.902},{"text":"messages","s":40.942,"e":41.422},{"text":"at","s":41.502,"e":41.662},{"text":"ninety","s":41.822,"e":42.062},{"text":"two","s":42.222,"e":42.302},{"text":"percent.","s":42.382,"e":43.262},{"text":"Above","s":43.422,"e":43.582},{"text":"point","s":43.662,"e":43.902},{"text":"nine","s":44.022,"e":44.142},{"text":"nine:","s":44.222,"e":44.942},{"text":"half","s":45.022,"e":45.262},{"text":"of","s":45.342,"e":45.502},{"text":"all","s":45.582,"e":45.742},{"text":"messages,","s":45.822,"e":46.542},{"text":"at","s":46.622,"e":46.702},{"text":"ninety","s":46.862,"e":47.102},{"text":"six","s":47.262,"e":47.422},{"text":"percent.","s":47.502,"e":48.302},{"text":"To","s":48.382,"e":48.462},{"text":"hit","s":48.542,"e":48.622},{"text":"ninety","s":48.662,"e":48.862},{"text":"five,","s":49.022,"e":49.582},{"text":"you","s":49.622,"e":49.662},{"text":"can","s":49.742,"e":49.822},{"text":"automate","s":49.902,"e":50.302},{"text":"about","s":50.382,"e":50.702},{"text":"fifty","s":50.782,"e":51.022},{"text":"seven","s":51.182,"e":51.342},{"text":"percent.","s":51.502,"e":52.142}]},{"template":"list","chapter":"Picking a threshold","narration":"How do you find your curve? Shreya Shankar, who researches LLM pipelines, uses a simple recipe. Take a sample of real examples; she uses five hundred. Label them. Run the model and record its probabilities. Then simulate every cutoff, and pick the one that meets your accuracy target with the most automation. It's an estimate from a sample, so leave some margin.","params":{"title":"Pick the threshold from data","style":"num","items":[{"text":"Sample real examples (she uses **500**)","at":"a sample of real examples"},{"text":"Label them (people, or a careful reasoning model)","at":"Label them"},{"text":"Run the model; record each probability","at":"record its probabilities"},{"text":"Simulate every cutoff: share automated vs accuracy","at":"simulate every cutoff"},{"text":"Pick the most automation that meets your target, with margin","at":"leave some margin"}]},"start":52.763,"dur":25.479,"words":[{"text":"How","s":53.063,"e":53.303},{"text":"do","s":53.343,"e":53.383},{"text":"you","s":53.463,"e":53.623},{"text":"find","s":53.703,"e":53.943},{"text":"your","s":54.063,"e":54.183},{"text":"curve?","s":54.463,"e":55.303},{"text":"Shreya","s":55.383,"e":55.703},{"text":"Shankar,","s":55.783,"e":56.423},{"text":"who","s":56.503,"e":56.583},{"text":"researches","s":56.783,"e":57.143},{"text":"LLM","s":57.223,"e":57.623},{"text":"pipelines,","s":57.703,"e":58.423},{"text":"uses","s":58.623,"e":58.823},{"text":"a","s":58.903,"e":58.983},{"text":"simple","s":59.063,"e":59.303},{"text":"recipe.","s":59.383,"e":60.343},{"text":"Take","s":60.423,"e":60.503},{"text":"a","s":60.583,"e":60.663},{"text":"sample","s":60.863,"e":61.063},{"text":"of","s":61.143,"e":61.223},{"text":"real","s":61.303,"e":61.543},{"text":"examples;","s":61.903,"e":62.583},{"text":"she","s":62.663,"e":62.823},{"text":"uses","s":62.983,"e":63.143},{"text":"five","s":63.223,"e":63.383},{"text":"hundred.","s":63.463,"e":64.343},{"text":"Label","s":64.423,"e":64.663},{"text":"them.","s":64.743,"e":65.463},{"text":"Run","s":65.543,"e":65.703},{"text":"the","s":65.743,"e":65.783},{"text":"model","s":65.863,"e":66.263},{"text":"and","s":66.343,"e":66.423},{"text":"record","s":66.583,"e":66.743},{"text":"its","s":66.823,"e":66.903},{"text":"probabilities.","s":66.983,"e":68.183},{"text":"Then","s":68.263,"e":68.583},{"text":"simulate","s":68.823,"e":69.303},{"text":"every","s":69.383,"e":69.703},{"text":"cutoff,","s":69.743,"e":70.263},{"text":"and","s":70.343,"e":70.423},{"text":"pick","s":70.543,"e":70.663},{"text":"the","s":70.703,"e":70.743},{"text":"one","s":70.823,"e":70.983},{"text":"that","s":71.063,"e":71.143},{"text":"meets","s":71.183,"e":71.303},{"text":"your","s":71.383,"e":71.543},{"text":"accuracy","s":71.783,"e":72.023},{"text":"target","s":72.103,"e":72.503},{"text":"with","s":72.583,"e":72.663},{"text":"the","s":72.703,"e":72.743},{"text":"most","s":72.903,"e":73.223},{"text":"automation.","s":73.303,"e":74.423},{"text":"It's","s":74.503,"e":74.623},{"text":"an","s":74.663,"e":74.743},{"text":"estimate","s":74.943,"e":75.143},{"text":"from","s":75.223,"e":75.303},{"text":"a","s":75.383,"e":75.463},{"text":"sample,","s":75.663,"e":76.023},{"text":"so","s":76.103,"e":76.263},{"text":"leave","s":76.503,"e":76.663},{"text":"some","s":76.743,"e":76.823},{"text":"margin.","s":77.063,"e":77.623}]},{"template":"stat","chapter":"Review the doubtful first","narration":"Probabilities also tell people where to look. Suppose reviewers fix every answer they check. To get the whole system to ninety five percent, reviewing a random sample means checking three out of every four messages. Reviewing the least confident first, it's one in three.","params":{"title":"Human review needed to reach 95% overall (BANKING77, Jev 1.13)","items":[{"value":75,"suffix":"%","label":"random review","tone":"bad","at":"random sample"},{"value":33,"suffix":"%","label":"least confident first","tone":"good","at":"least confident first","hero":true}],"src":"Computed from our 3,080 answers, assuming reviewers are always right. To reach 90%: 19% vs 50%."},"start":78.242,"dur":17.956,"words":[{"text":"Probabilities","s":78.542,"e":79.342},{"text":"also","s":79.502,"e":79.822},{"text":"tell","s":79.902,"e":80.062},{"text":"people","s":80.222,"e":80.382},{"text":"where","s":80.542,"e":80.782},{"text":"to","s":80.862,"e":80.942},{"text":"look.","s":81.022,"e":81.582},{"text":"Suppose","s":81.822,"e":82.142},{"text":"reviewers","s":82.222,"e":82.702},{"text":"fix","s":82.782,"e":83.022},{"text":"every","s":83.182,"e":83.422},{"text":"answer","s":83.502,"e":83.822},{"text":"they","s":83.902,"e":83.982},{"text":"check.","s":84.062,"e":84.582},{"text":"To","s":84.782,"e":84.862},{"text":"get","s":84.902,"e":84.942},{"text":"the","s":85.022,"e":85.102},{"text":"whole","s":85.182,"e":85.342},{"text":"system","s":85.422,"e":85.822},{"text":"to","s":85.902,"e":86.062},{"text":"ninety","s":86.102,"e":86.302},{"text":"five","s":86.382,"e":86.542},{"text":"percent,","s":86.622,"e":87.422},{"text":"reviewing","s":87.582,"e":87.822},{"text":"a","s":87.902,"e":87.982},{"text":"random","s":88.062,"e":88.382},{"text":"sample","s":88.462,"e":88.862},{"text":"means","s":88.942,"e":89.182},{"text":"checking","s":89.262,"e":89.582},{"text":"three","s":89.742,"e":90.062},{"text":"out","s":90.142,"e":90.222},{"text":"of","s":90.302,"e":90.382},{"text":"every","s":90.542,"e":90.782},{"text":"four","s":90.942,"e":91.102},{"text":"messages.","s":91.182,"e":92.022},{"text":"Reviewing","s":92.222,"e":92.622},{"text":"the","s":92.662,"e":92.702},{"text":"least","s":92.862,"e":93.102},{"text":"confident","s":93.182,"e":93.582},{"text":"first,","s":93.662,"e":94.382},{"text":"it's","s":94.462,"e":94.622},{"text":"one","s":94.662,"e":94.702},{"text":"in","s":94.782,"e":94.942},{"text":"three.","s":95.102,"e":95.582}]},{"template":"statement","chapter":"Ranking beats accuracy","narration":"Which means the property that matters most isn't raw accuracy. It's whether wrong answers come with low probability. One independent test on the same banking set found a fine tuned model whose accuracy moved by under a point, while the share it could handle on its own jumped from sixty four to ninety two in a hundred.","params":{"eyebrow":"Model Fatigue, testing open decision models on BANKING77","text":"Accuracy moved *under a point*. Automation went from *64 to 92* in a hundred.","sub":"What matters is whether wrong answers come with low probability.","sub_at":"wrong answers","art":"chart","tone":"agent","punch":"sixty four to ninety two"},"start":96.198,"dur":20.359,"words":[{"text":"Which","s":96.498,"e":96.738},{"text":"means","s":96.858,"e":96.978},{"text":"the","s":97.058,"e":97.138},{"text":"property","s":97.218,"e":97.618},{"text":"that","s":97.698,"e":97.778},{"text":"matters","s":97.911,"e":98.178},{"text":"most","s":98.418,"e":98.658},{"text":"isn't","s":98.818,"e":99.218},{"text":"raw","s":99.378,"e":99.538},{"text":"accuracy.","s":99.898,"e":100.738},{"text":"It's","s":100.818,"e":100.938},{"text":"whether","s":100.978,"e":101.218},{"text":"wrong","s":101.298,"e":101.538},{"text":"answers","s":101.618,"e":102.018},{"text":"come","s":102.098,"e":102.258},{"text":"with","s":102.378,"e":102.498},{"text":"low","s":102.578,"e":102.738},{"text":"probability.","s":102.818,"e":104.058},{"text":"One","s":104.098,"e":104.258},{"text":"independent","s":104.338,"e":104.818},{"text":"test","s":104.898,"e":105.218},{"text":"on","s":105.258,"e":105.298},{"text":"the","s":105.378,"e":105.538},{"text":"same","s":105.658,"e":105.778},{"text":"banking","s":105.858,"e":106.258},{"text":"set","s":106.418,"e":106.578},{"text":"found","s":106.698,"e":106.818},{"text":"a","s":106.898,"e":106.978},{"text":"fine","s":107.058,"e":107.298},{"text":"tuned","s":107.378,"e":107.618},{"text":"model","s":107.778,"e":107.938},{"text":"whose","s":108.058,"e":108.178},{"text":"accuracy","s":108.458,"e":108.738},{"text":"moved","s":108.818,"e":109.058},{"text":"by","s":109.178,"e":109.298},{"text":"under","s":109.378,"e":109.618},{"text":"a","s":109.698,"e":109.778},{"text":"point,","s":109.858,"e":110.578},{"text":"while","s":110.658,"e":110.738},{"text":"the","s":110.818,"e":110.898},{"text":"share","s":110.978,"e":111.058},{"text":"it","s":111.138,"e":111.218},{"text":"could","s":111.298,"e":111.378},{"text":"handle","s":111.498,"e":111.618},{"text":"on","s":111.698,"e":111.778},{"text":"its","s":111.898,"e":112.018},{"text":"own","s":112.258,"e":112.498},{"text":"jumped","s":112.578,"e":112.738},{"text":"from","s":112.818,"e":113.058},{"text":"sixty","s":113.178,"e":113.458},{"text":"four","s":113.618,"e":113.778},{"text":"to","s":113.858,"e":114.098},{"text":"ninety","s":114.178,"e":114.578},{"text":"two","s":114.778,"e":114.978},{"text":"in","s":115.018,"e":115.058},{"text":"a","s":115.098,"e":115.138},{"text":"hundred.","s":115.218,"e":115.938}]},{"template":"compare","chapter":"Answer less precisely","narration":"There's another option besides asking a person: answer less precisely. TypeSafe classified sixty company filings into seventy five industry groups. Where the model was sure, it was right twenty seven times out of thirty. Where it wasn't, only twelve. But reporting those uncertain ones at the broader division level took them to seventy percent. Same request, no second call.","params":{"title":"60 SEC filings, 75 industry groups","left":{"title":"Sure (confidence ≥ 0.9)","tone":"good","items":[{"text":"**27/30** right at group level","at":"twenty seven"}],"who":"snap"},"right":{"title":"Unsure (< 0.9)","tone":"warn","items":[{"text":"**12/30** right as a group","at":"only twelve"},{"text":"**21/30** right as a division","at":"seventy percent"}],"who":"snap"},"vs":"vs","hnote":{"text":"48 of 60 useful answers","at":"Same request","p":[50,77],"pp":[50,76]}},"start":116.557,"dur":24.669,"words":[{"text":"There's","s":116.857,"e":117.097},{"text":"another","s":117.257,"e":117.417},{"text":"option","s":117.497,"e":117.817},{"text":"besides","s":117.937,"e":118.217},{"text":"asking","s":118.297,"e":118.617},{"text":"a","s":118.657,"e":118.697},{"text":"person:","s":118.777,"e":119.657},{"text":"answer","s":119.737,"e":120.137},{"text":"less","s":120.297,"e":120.537},{"text":"precisely.","s":120.617,"e":121.337},{"text":"TypeSafe","s":121.897,"e":122.457},{"text":"classified","s":122.537,"e":123.097},{"text":"sixty","s":123.257,"e":123.577},{"text":"company","s":123.657,"e":123.897},{"text":"filings","s":124.057,"e":124.537},{"text":"into","s":124.617,"e":124.857},{"text":"seventy","s":124.937,"e":125.337},{"text":"five","s":125.417,"e":125.657},{"text":"industry","s":125.737,"e":126.137},{"text":"groups.","s":126.217,"e":126.697},{"text":"Where","s":127.097,"e":127.257},{"text":"the","s":127.297,"e":127.337},{"text":"model","s":127.417,"e":127.577},{"text":"was","s":127.657,"e":127.817},{"text":"sure,","s":127.897,"e":128.457},{"text":"it","s":128.497,"e":128.537},{"text":"was","s":128.617,"e":128.697},{"text":"right","s":128.777,"e":129.017},{"text":"twenty","s":129.097,"e":129.337},{"text":"seven","s":129.417,"e":129.577},{"text":"times","s":129.657,"e":129.977},{"text":"out","s":130.017,"e":130.057},{"text":"of","s":130.137,"e":130.217},{"text":"thirty.","s":130.297,"e":131.017},{"text":"Where","s":131.097,"e":131.177},{"text":"it","s":131.257,"e":131.337},{"text":"wasn't,","s":131.457,"e":132.057},{"text":"only","s":132.217,"e":132.377},{"text":"twelve.","s":132.457,"e":133.177},{"text":"But","s":133.417,"e":133.497},{"text":"reporting","s":133.657,"e":133.977},{"text":"those","s":134.057,"e":134.137},{"text":"uncertain","s":134.217,"e":134.697},{"text":"ones","s":134.777,"e":135.017},{"text":"at","s":135.057,"e":135.097},{"text":"the","s":135.177,"e":135.257},{"text":"broader","s":135.337,"e":135.577},{"text":"division","s":135.777,"e":135.977},{"text":"level","s":136.057,"e":136.377},{"text":"took","s":136.537,"e":136.697},{"text":"them","s":136.777,"e":136.857},{"text":"to","s":136.977,"e":137.097},{"text":"seventy","s":137.137,"e":137.457},{"text":"percent.","s":137.497,"e":138.457},{"text":"Same","s":138.577,"e":138.697},{"text":"request,","s":138.977,"e":139.497},{"text":"no","s":139.577,"e":139.657},{"text":"second","s":139.737,"e":139.977},{"text":"call.","s":140.057,"e":140.617}]},{"template":"statement","chapter":"Honest uncertainty","narration":"This is the deeper point of the whole module. The TypeSafe docs put it plainly: if an intelligent system, whether human or machine, cannot express honest uncertainty, the system cannot be trusted. I don't know is a feature. Build paths for it.","params":{"eyebrow":"TypeSafe docs, Confidence","text":"If a system can't express *honest uncertainty*, it can't be trusted.","art":"snap","mood":"unsure","react":{"at":"I don't know is a feature","mood":"happy"},"punch":"honest uncertainty"},"start":141.226,"dur":18.112,"words":[{"text":"This","s":141.526,"e":141.766},{"text":"is","s":141.846,"e":141.926},{"text":"the","s":142.006,"e":142.086},{"text":"deeper","s":142.166,"e":142.486},{"text":"point","s":142.606,"e":142.726},{"text":"of","s":142.806,"e":142.886},{"text":"the","s":142.966,"e":143.046},{"text":"whole","s":143.126,"e":143.366},{"text":"module.","s":143.446,"e":144.406},{"text":"The","s":144.486,"e":144.566},{"text":"TypeSafe","s":144.686,"e":145.046},{"text":"docs","s":145.126,"e":145.446},{"text":"put","s":145.526,"e":145.606},{"text":"it","s":145.686,"e":145.766},{"text":"plainly:","s":145.926,"e":146.646},{"text":"if","s":146.726,"e":146.806},{"text":"an","s":146.846,"e":146.886},{"text":"intelligent","s":146.966,"e":147.446},{"text":"system,","s":147.526,"e":148.246},{"text":"whether","s":148.366,"e":148.486},{"text":"human","s":148.566,"e":148.966},{"text":"or","s":149.046,"e":149.206},{"text":"machine,","s":149.486,"e":150.086},{"text":"cannot","s":150.326,"e":150.566},{"text":"express","s":150.646,"e":151.046},{"text":"honest","s":151.206,"e":151.526},{"text":"uncertainty,","s":151.606,"e":152.646},{"text":"the","s":152.726,"e":152.806},{"text":"system","s":153.046,"e":153.286},{"text":"cannot","s":153.366,"e":153.846},{"text":"be","s":153.886,"e":153.926},{"text":"trusted.","s":154.006,"e":155.046},{"text":"I","s":155.166,"e":155.286},{"text":"don't","s":155.326,"e":155.526},{"text":"know","s":155.606,"e":155.926},{"text":"is","s":156.086,"e":156.246},{"text":"a","s":156.326,"e":156.406},{"text":"feature.","s":156.486,"e":157.166},{"text":"Build","s":157.366,"e":157.606},{"text":"paths","s":157.686,"e":158.006},{"text":"for","s":158.086,"e":158.246},{"text":"it.","s":158.326,"e":158.726}]},{"template":"statement","chapter":"Rare cases","narration":"One trap. When the thing you're looking for is rare, accuracy lies. If one percent of messages report fraud, a rule that never flags anything is ninety nine percent accurate, and useless. For rare classes, judge thresholds by precision and recall on the flagged items, and count how many a person must read.","params":{"eyebrow":"The rare-positive trap","text":"If 1% is fraud, “never flag” is *99% accurate* and useless.","sub":"Judge rare classes by precision and recall, and by the review workload they create.","sub_at":"precision and recall","art":"warn","tone":"bad","punch":"useless"},"start":159.338,"dur":22.188,"words":[{"text":"One","s":159.638,"e":159.878},{"text":"trap.","s":159.958,"e":160.438},{"text":"When","s":160.918,"e":161.078},{"text":"the","s":161.118,"e":161.158},{"text":"thing","s":161.238,"e":161.398},{"text":"you're","s":161.438,"e":161.558},{"text":"looking","s":161.638,"e":161.878},{"text":"for","s":161.958,"e":162.118},{"text":"is","s":162.198,"e":162.278},{"text":"rare,","s":162.358,"e":163.078},{"text":"accuracy","s":163.158,"e":163.798},{"text":"lies.","s":163.958,"e":164.758},{"text":"If","s":164.998,"e":165.158},{"text":"one","s":165.238,"e":165.398},{"text":"percent","s":165.478,"e":165.798},{"text":"of","s":165.838,"e":165.878},{"text":"messages","s":165.958,"e":166.358},{"text":"report","s":166.438,"e":166.758},{"text":"fraud,","s":166.838,"e":167.558},{"text":"a","s":167.638,"e":167.718},{"text":"rule","s":167.798,"e":167.958},{"text":"that","s":168.038,"e":168.198},{"text":"never","s":168.358,"e":168.598},{"text":"flags","s":168.678,"e":168.998},{"text":"anything","s":169.158,"e":169.638},{"text":"is","s":169.798,"e":169.958},{"text":"ninety","s":170.118,"e":170.358},{"text":"nine","s":170.438,"e":170.518},{"text":"percent","s":170.678,"e":170.918},{"text":"accurate,","s":171.078,"e":171.798},{"text":"and","s":171.918,"e":172.038},{"text":"useless.","s":172.198,"e":173.038},{"text":"For","s":173.318,"e":173.478},{"text":"rare","s":173.518,"e":173.718},{"text":"classes,","s":173.798,"e":174.598},{"text":"judge","s":174.758,"e":174.918},{"text":"thresholds","s":174.998,"e":175.478},{"text":"by","s":175.558,"e":175.638},{"text":"precision","s":175.718,"e":176.358},{"text":"and","s":176.438,"e":176.518},{"text":"recall","s":176.598,"e":177.078},{"text":"on","s":177.118,"e":177.158},{"text":"the","s":177.238,"e":177.318},{"text":"flagged","s":177.398,"e":177.718},{"text":"items,","s":177.798,"e":178.518},{"text":"and","s":178.598,"e":178.678},{"text":"count","s":178.758,"e":178.998},{"text":"how","s":179.078,"e":179.158},{"text":"many","s":179.318,"e":179.558},{"text":"a","s":179.598,"e":179.638},{"text":"person","s":179.718,"e":179.958},{"text":"must","s":180.038,"e":180.198},{"text":"read.","s":180.278,"e":180.918}]},{"template":"quiz","chapter":"Quick check","narration":"Quick check. On your labeled sample, answers above point nine five are ninety seven percent right and cover sixty percent of traffic. Your target is ninety five percent. What do you do with the other forty percent?","params":{"q":"≥ 0.95 covers **60%** of traffic at **97%**. Target is 95%. The other 40%?","options":["Automate them too; the average is still fine","Route them to review, least confident first, or answer them less precisely","Drop them","Lower the threshold until everything is automated"],"answer":1,"explain":"Automate the sure 60%. Send the rest to people (least confident first) or to a coarser answer, and re-measure as traffic changes."},"start":181.526,"dur":18.587,"words":[{"text":"Quick","s":181.826,"e":182.066},{"text":"check.","s":182.146,"e":182.666},{"text":"On","s":182.866,"e":182.946},{"text":"your","s":183.026,"e":183.106},{"text":"labeled","s":183.186,"e":183.506},{"text":"sample,","s":183.586,"e":184.306},{"text":"answers","s":184.506,"e":184.706},{"text":"above","s":184.866,"e":185.026},{"text":"point","s":185.106,"e":185.346},{"text":"nine","s":185.426,"e":185.746},{"text":"five","s":185.826,"e":186.146},{"text":"are","s":186.226,"e":186.466},{"text":"ninety","s":186.506,"e":186.786},{"text":"seven","s":186.866,"e":187.026},{"text":"percent","s":187.106,"e":187.426},{"text":"right","s":187.506,"e":187.826},{"text":"and","s":187.906,"e":187.986},{"text":"cover","s":188.066,"e":188.386},{"text":"sixty","s":188.466,"e":188.786},{"text":"percent","s":188.866,"e":189.106},{"text":"of","s":189.146,"e":189.186},{"text":"traffic.","s":189.266,"e":190.226},{"text":"Your","s":190.306,"e":190.386},{"text":"target","s":190.466,"e":190.866},{"text":"is","s":190.946,"e":191.106},{"text":"ninety","s":191.146,"e":191.346},{"text":"five","s":191.506,"e":191.666},{"text":"percent.","s":191.746,"e":192.706},{"text":"What","s":192.786,"e":192.866},{"text":"do","s":192.906,"e":192.946},{"text":"you","s":193.026,"e":193.106},{"text":"do","s":193.146,"e":193.186},{"text":"with","s":193.266,"e":193.346},{"text":"the","s":193.426,"e":193.506},{"text":"other","s":193.666,"e":193.826},{"text":"forty","s":193.946,"e":194.106},{"text":"percent?","s":194.146,"e":194.946}]},{"template":"exercise","chapter":"Try it","narration":"Try it. Have your agent build a coverage table for one of your decisions, and turn it into a threshold and a review budget.","params":{"title":"Build your coverage table","intro":"Paste this into your coding agent:","prompt":"Using 300 labeled examples of one decision, run the model and record the\nprobability of its answer. For thresholds 0, 0.5, 0.7, 0.8, 0.9, 0.95,\n0.99, print: share automated, accuracy on the automated share, and how\nmany items per day would go to review at our current volume. Recommend\nthe threshold that meets a 95% target with the most automation, and\nexplain the margin you left. If there's a parent category, also report\naccuracy when unsure answers fall back to it."},"start":200.113,"dur":7.872,"words":[{"text":"Try","s":200.413,"e":200.653},{"text":"it.","s":200.733,"e":200.973},{"text":"Have","s":201.293,"e":201.453},{"text":"your","s":201.533,"e":201.613},{"text":"agent","s":201.773,"e":201.933},{"text":"build","s":202.013,"e":202.173},{"text":"a","s":202.213,"e":202.253},{"text":"coverage","s":202.333,"e":202.653},{"text":"table","s":202.733,"e":203.053},{"text":"for","s":203.133,"e":203.213},{"text":"one","s":203.293,"e":203.373},{"text":"of","s":203.413,"e":203.453},{"text":"your","s":203.533,"e":203.613},{"text":"decisions,","s":203.693,"e":204.333},{"text":"and","s":204.453,"e":204.573},{"text":"turn","s":204.733,"e":204.893},{"text":"it","s":204.973,"e":205.013},{"text":"into","s":205.053,"e":205.213},{"text":"a","s":205.293,"e":205.373},{"text":"threshold","s":205.453,"e":206.013},{"text":"and","s":206.093,"e":206.253},{"text":"a","s":206.333,"e":206.373},{"text":"review","s":206.413,"e":206.653},{"text":"budget.","s":206.733,"e":207.373}]},{"template":"recap","chapter":"Recap","narration":"So: ask how much you can automate at the accuracy you need, and read it off a coverage curve built from your own labels. Review the doubtful first. Answer less precisely when you can. And watch out for rare classes. That's module three. Next: composing many small judgments.","params":{"items":["Ask: what share can we automate at the accuracy we need?","Read it off a coverage curve from 200–500 labels.","Review least confident first: 33% vs 75% to reach 95%.","Fall back to coarser answers; judge rare classes by precision."],"next":"Decompose: many small questions","next_at":"Next"},"start":207.985,"dur":21.309,"words":[{"text":"So:","s":208.285,"e":208.845},{"text":"ask","s":209.085,"e":209.325},{"text":"how","s":209.405,"e":209.485},{"text":"much","s":209.565,"e":209.725},{"text":"you","s":209.765,"e":209.805},{"text":"can","s":209.885,"e":210.045},{"text":"automate","s":210.205,"e":210.525},{"text":"at","s":210.605,"e":210.685},{"text":"the","s":210.765,"e":210.845},{"text":"accuracy","s":211.165,"e":211.485},{"text":"you","s":211.565,"e":211.645},{"text":"need,","s":211.725,"e":212.285},{"text":"and","s":212.405,"e":212.525},{"text":"read","s":212.605,"e":212.685},{"text":"it","s":212.725,"e":212.765},{"text":"off","s":212.845,"e":212.925},{"text":"a","s":213.005,"e":213.085},{"text":"coverage","s":213.285,"e":213.485},{"text":"curve","s":213.565,"e":213.885},{"text":"built","s":214.005,"e":214.125},{"text":"from","s":214.205,"e":214.285},{"text":"your","s":214.405,"e":214.525},{"text":"own","s":214.645,"e":214.765},{"text":"labels.","s":215.045,"e":215.805},{"text":"Review","s":216.005,"e":216.205},{"text":"the","s":216.285,"e":216.365},{"text":"doubtful","s":216.445,"e":216.845},{"text":"first.","s":217.005,"e":217.885},{"text":"Answer","s":218.085,"e":218.285},{"text":"less","s":218.365,"e":218.605},{"text":"precisely","s":218.685,"e":219.245},{"text":"when","s":219.325,"e":219.405},{"text":"you","s":219.485,"e":219.565},{"text":"can.","s":219.645,"e":220.365},{"text":"And","s":220.485,"e":220.605},{"text":"watch","s":220.725,"e":220.845},{"text":"out","s":220.925,"e":221.085},{"text":"for","s":221.165,"e":221.245},{"text":"rare","s":221.405,"e":221.645},{"text":"classes.","s":221.725,"e":222.765},{"text":"That's","s":222.885,"e":223.085},{"text":"module","s":223.165,"e":223.485},{"text":"three.","s":223.645,"e":224.445},{"text":"Next:","s":224.525,"e":225.245},{"text":"composing","s":225.325,"e":225.805},{"text":"many","s":225.965,"e":226.205},{"text":"small","s":226.365,"e":226.685},{"text":"judgments.","s":226.765,"e":227.725}]}],"total":229.694,"audio":"/audio/11-automate-the-sure-ones.mp3"}