{"id":"19-jagged-edges","number":19,"slug":"jagged-edges","title":"Know the jagged edges","short":"Jagged edges","module":"Shipping it","summary":"Decision models are strong at semantic judgment and unreliable at things a calculator does. Here are the failure modes we reproduced (counting, arithmetic, dates), TypeSafe's own list of nine, the cases where the model is wrong without sounding unsure, and the rule that fixes most of them: reason in code, decide with the model.","voice":"Jessica","scenes":[{"template":"title","chapter":"Intro","narration":"Every model has edges where it suddenly gets worse. For decision models, those edges are sharp, and easy to predict. Let's walk right up to them.","params":{"kicker":"Lesson 19","module":"Module 6 · Shipping it","title":"Know the *jagged edges*","subtitle":"Brilliant at meaning. Bad at arithmetic.","cast":[{"who":"snap","mood":"unsure"},"curly"]},"start":0.5,"dur":10.604,"words":[{"text":"Every","s":0.8,"e":1.12},{"text":"model","s":1.2,"e":1.44},{"text":"has","s":1.52,"e":1.76},{"text":"edges","s":1.84,"e":2.24},{"text":"where","s":2.32,"e":2.4},{"text":"it","s":2.48,"e":2.56},{"text":"suddenly","s":2.72,"e":3.04},{"text":"gets","s":3.12,"e":3.28},{"text":"worse.","s":3.44,"e":4.16},{"text":"For","s":4.4,"e":4.48},{"text":"decision","s":4.68,"e":4.88},{"text":"models,","s":4.96,"e":5.6},{"text":"those","s":5.72,"e":5.84},{"text":"edges","s":5.92,"e":6.16},{"text":"are","s":6.24,"e":6.48},{"text":"sharp,","s":6.56,"e":7.04},{"text":"and","s":7.16,"e":7.28},{"text":"easy","s":7.36,"e":7.6},{"text":"to","s":7.68,"e":7.76},{"text":"predict.","s":8,"e":8.56},{"text":"Let's","s":8.64,"e":8.8},{"text":"walk","s":8.88,"e":9.04},{"text":"right","s":9.2,"e":9.36},{"text":"up","s":9.44,"e":9.52},{"text":"to","s":9.6,"e":9.68},{"text":"them.","s":9.8,"e":10.24}]},{"template":"quote","chapter":"Jagged","narration":"TypeSafe's CEO doesn't hide it. Our model cannot count R's in strawberry, he says. Luckily, that is not what real world work looks like. Most work is semantic judgment. But you need to know exactly where the edges are.","params":{"text":"Our model cannot count R's in strawberry. *Luckily, that is not what real world work looks like.*","who":"Diogo Almeida","role":"TypeSafe AI","src":"The TWIML AI Podcast","t":2070},"start":11.104,"dur":15.788,"words":[{"text":"TypeSafe's","s":11.404,"e":12.044},{"text":"CEO","s":12.124,"e":12.604},{"text":"doesn't","s":12.844,"e":13.084},{"text":"hide","s":13.164,"e":13.324},{"text":"it.","s":13.404,"e":13.964},{"text":"Our","s":14.084,"e":14.204},{"text":"model","s":14.404,"e":14.604},{"text":"cannot","s":14.684,"e":15.164},{"text":"count","s":15.244,"e":15.484},{"text":"R's","s":15.564,"e":15.804},{"text":"in","s":15.884,"e":15.964},{"text":"strawberry,","s":16.044,"e":16.844},{"text":"he","s":16.924,"e":17.004},{"text":"says.","s":17.204,"e":17.644},{"text":"Luckily,","s":17.884,"e":18.524},{"text":"that","s":18.604,"e":18.684},{"text":"is","s":18.804,"e":18.924},{"text":"not","s":19.004,"e":19.244},{"text":"what","s":19.324,"e":19.404},{"text":"real","s":19.484,"e":19.644},{"text":"world","s":19.724,"e":19.964},{"text":"work","s":20.044,"e":20.284},{"text":"looks","s":20.404,"e":20.524},{"text":"like.","s":20.604,"e":21.244},{"text":"Most","s":21.404,"e":21.564},{"text":"work","s":21.684,"e":21.804},{"text":"is","s":21.884,"e":22.044},{"text":"semantic","s":22.324,"e":22.604},{"text":"judgment.","s":22.684,"e":23.564},{"text":"But","s":23.644,"e":23.724},{"text":"you","s":23.804,"e":23.884},{"text":"need","s":23.964,"e":24.044},{"text":"to","s":24.124,"e":24.204},{"text":"know","s":24.324,"e":24.444},{"text":"exactly","s":24.644,"e":25.084},{"text":"where","s":25.164,"e":25.244},{"text":"the","s":25.324,"e":25.364},{"text":"edges","s":25.404,"e":25.724},{"text":"are.","s":25.804,"e":26.284}]},{"template":"jagged","chapter":"Counting","narration":"We checked. Twelve words, six of them fruits. Asked as one question, how many are fruits, the model said five, at fifty two percent. Wrong. Now ask twelve small Nouls, one per word: is this a fruit? Every one came back right, and code adds them up. Six.","params":{"kicker":"Counting","state":"[apple, typesafe, california, banana, likes, calibration, orange, vertex, mango, kiwi, router, plum]","type":"choice","question":"How many entries in `items` are names of fruits?","answer":{"text":"5","p":0.52},"truth":"6","answer_at":"the model said five","truth_at":"Wrong","fix":{"label":"one Noul per item, sum in code: 6","lang":"py","code":"count = sum(a[f\"item_{i}\"].noul > 0.5 for i in range(len(items)))"},"fix_at":"Now ask twelve"},"start":26.892,"dur":20.176,"words":[{"text":"We","s":27.192,"e":27.352},{"text":"checked.","s":27.552,"e":27.912},{"text":"Twelve","s":28.072,"e":28.312},{"text":"words,","s":28.392,"e":29.032},{"text":"six","s":29.112,"e":29.272},{"text":"of","s":29.352,"e":29.432},{"text":"them","s":29.512,"e":29.592},{"text":"fruits.","s":29.752,"e":30.392},{"text":"Asked","s":30.552,"e":30.872},{"text":"as","s":30.952,"e":31.032},{"text":"one","s":31.192,"e":31.352},{"text":"question,","s":31.432,"e":32.232},{"text":"how","s":32.312,"e":32.392},{"text":"many","s":32.472,"e":32.632},{"text":"are","s":32.712,"e":32.792},{"text":"fruits,","s":32.872,"e":33.592},{"text":"the","s":33.632,"e":33.672},{"text":"model","s":33.752,"e":33.992},{"text":"said","s":34.072,"e":34.312},{"text":"five,","s":34.472,"e":34.952},{"text":"at","s":35.032,"e":35.192},{"text":"fifty","s":35.272,"e":35.512},{"text":"two","s":35.592,"e":35.672},{"text":"percent.","s":35.832,"e":36.712},{"text":"Wrong.","s":36.792,"e":37.512},{"text":"Now","s":37.752,"e":37.912},{"text":"ask","s":37.992,"e":38.312},{"text":"twelve","s":38.392,"e":38.712},{"text":"small","s":38.872,"e":39.192},{"text":"Nouls,","s":39.272,"e":39.912},{"text":"one","s":39.992,"e":40.152},{"text":"per","s":40.232,"e":40.312},{"text":"word:","s":40.392,"e":41.112},{"text":"is","s":41.192,"e":41.272},{"text":"this","s":41.352,"e":41.432},{"text":"a","s":41.512,"e":41.552},{"text":"fruit?","s":41.592,"e":42.312},{"text":"Every","s":42.392,"e":42.712},{"text":"one","s":42.872,"e":43.112},{"text":"came","s":43.192,"e":43.352},{"text":"back","s":43.432,"e":43.592},{"text":"right,","s":43.672,"e":44.152},{"text":"and","s":44.232,"e":44.312},{"text":"code","s":44.472,"e":44.632},{"text":"adds","s":44.712,"e":44.872},{"text":"them","s":44.952,"e":45.032},{"text":"up.","s":45.112,"e":45.752},{"text":"Six.","s":45.992,"e":46.472}]},{"template":"jagged","chapter":"Arithmetic","narration":"Arithmetic. An insurance claim lists four costs adding up to three thousand two hundred fifty, but claims three thousand one fifty. Do the costs add up? The model said point four three. That's honest, it really doesn't know, but it's useless. One line of code knows for sure.","params":{"kicker":"Arithmetic","state":"costs: 1,700 + 800 + 450 + 300 · total_claimed: 3,150","type":"noul","question":"Do the costs in `line_items` add up exactly to `total_claimed`?","answer":{"text":"unsure","p":0.43},"truth":"no (3,250)","answer_at":"point four three","truth_at":"useless","fix":{"label":"keep the arithmetic in code","lang":"py","code":"adds_up = sum(i[\"cost\"] for i in claim[\"line_items\"]) == claim[\"total_claimed\"]"},"fix_at":"One line of code"},"start":47.068,"dur":20.672,"words":[{"text":"Arithmetic.","s":47.368,"e":48.168},{"text":"An","s":48.248,"e":48.688},{"text":"insurance","s":48.728,"e":49.235},{"text":"claim","s":49.288,"e":49.624},{"text":"lists","s":49.688,"e":50.048},{"text":"four","s":50.088,"e":50.348},{"text":"costs","s":50.408,"e":50.968},{"text":"adding","s":51.048,"e":51.288},{"text":"up","s":51.368,"e":51.448},{"text":"to","s":51.528,"e":51.688},{"text":"three","s":51.768,"e":51.848},{"text":"thousand","s":52.008,"e":52.328},{"text":"two","s":52.488,"e":52.568},{"text":"hundred","s":52.648,"e":52.888},{"text":"fifty,","s":53.048,"e":53.848},{"text":"but","s":53.928,"e":54.008},{"text":"claims","s":54.088,"e":54.488},{"text":"three","s":54.648,"e":54.808},{"text":"thousand","s":54.888,"e":55.208},{"text":"one","s":55.368,"e":55.528},{"text":"fifty.","s":55.608,"e":56.568},{"text":"Do","s":56.648,"e":56.728},{"text":"the","s":56.768,"e":56.808},{"text":"costs","s":56.888,"e":57.208},{"text":"add","s":57.288,"e":57.368},{"text":"up?","s":57.448,"e":57.768},{"text":"The","s":58.248,"e":58.328},{"text":"model","s":58.488,"e":58.648},{"text":"said","s":58.728,"e":58.968},{"text":"point","s":59.128,"e":59.368},{"text":"four","s":59.528,"e":59.768},{"text":"three.","s":59.848,"e":60.648},{"text":"That's","s":60.728,"e":60.968},{"text":"honest,","s":61.048,"e":61.688},{"text":"it","s":61.768,"e":61.848},{"text":"really","s":61.928,"e":62.168},{"text":"doesn't","s":62.248,"e":62.408},{"text":"know,","s":62.488,"e":63.208},{"text":"but","s":63.288,"e":63.528},{"text":"it's","s":63.608,"e":63.768},{"text":"useless.","s":63.848,"e":64.888},{"text":"One","s":65.008,"e":65.128},{"text":"line","s":65.208,"e":65.288},{"text":"of","s":65.368,"e":65.448},{"text":"code","s":65.528,"e":65.928},{"text":"knows","s":66.008,"e":66.248},{"text":"for","s":66.328,"e":66.408},{"text":"sure.","s":66.568,"e":67.128}]},{"template":"jagged","chapter":"Dates","narration":"Dates. An invoice from February tenth, due in thirty days, was paid on March fourteenth. Was it on time? Point four seven, a shrug. TypeSafe's docs explain why: the model reads dates as text, not as quantities. Extract the parts, as in lesson fifteen, and let code count the days.","params":{"kicker":"Date comparison","state":"invoice_date: 2026-02-10 · paid_date: March 14, 2026 · terms: due within 30 days","type":"noul","question":"Was the invoice paid within the 30 days allowed by `terms`?","answer":{"text":"unsure","p":0.47},"truth":"no (32 days)","answer_at":"Point four seven","truth_at":"a shrug","fix":{"label":"extract the parts, compare in code","lang":"py","code":"on_time = (paid - invoiced).days <= 30   # 32 days: late"},"fix_at":"Extract the parts"},"start":67.74,"dur":20.986,"words":[{"text":"Dates.","s":68.04,"e":68.6},{"text":"An","s":69,"e":69.16},{"text":"invoice","s":69.24,"e":69.64},{"text":"from","s":69.72,"e":69.8},{"text":"February","s":69.88,"e":70.28},{"text":"tenth,","s":70.44,"e":71},{"text":"due","s":71.08,"e":71.24},{"text":"in","s":71.32,"e":71.48},{"text":"thirty","s":71.56,"e":71.8},{"text":"days,","s":71.88,"e":72.28},{"text":"was","s":72.36,"e":72.52},{"text":"paid","s":72.6,"e":72.84},{"text":"on","s":72.92,"e":73},{"text":"March","s":73.08,"e":73.32},{"text":"fourteenth.","s":73.4,"e":74.24},{"text":"Was","s":74.44,"e":74.6},{"text":"it","s":74.64,"e":74.68},{"text":"on","s":74.72,"e":74.76},{"text":"time?","s":74.84,"e":75.24},{"text":"Point","s":75.72,"e":75.88},{"text":"four","s":75.96,"e":76.2},{"text":"seven,","s":76.36,"e":76.92},{"text":"a","s":76.96,"e":77},{"text":"shrug.","s":77.08,"e":77.72},{"text":"TypeSafe's","s":77.96,"e":78.52},{"text":"docs","s":78.68,"e":78.84},{"text":"explain","s":79.04,"e":79.24},{"text":"why:","s":79.32,"e":80.12},{"text":"the","s":80.2,"e":80.24},{"text":"model","s":80.28,"e":80.6},{"text":"reads","s":80.68,"e":80.92},{"text":"dates","s":81,"e":81.32},{"text":"as","s":81.4,"e":81.48},{"text":"text,","s":81.64,"e":82.28},{"text":"not","s":82.36,"e":82.6},{"text":"as","s":82.68,"e":82.76},{"text":"quantities.","s":82.84,"e":83.68},{"text":"Extract","s":83.88,"e":84.28},{"text":"the","s":84.32,"e":84.36},{"text":"parts,","s":84.44,"e":84.92},{"text":"as","s":85,"e":85.08},{"text":"in","s":85.16,"e":85.24},{"text":"lesson","s":85.36,"e":85.48},{"text":"fifteen,","s":85.56,"e":86.28},{"text":"and","s":86.32,"e":86.36},{"text":"let","s":86.44,"e":86.6},{"text":"code","s":86.68,"e":87},{"text":"count","s":87.08,"e":87.24},{"text":"the","s":87.28,"e":87.32},{"text":"days.","s":87.4,"e":88.12}]},{"template":"table","chapter":"The full list","narration":"TypeSafe publishes the full list for Jev one point thirteen, nine failure modes, each with a fix. Literal reading. Math. Date comparison. Indirection. Large, noisy state. Adversarial content. Instructions that contradict the criteria. Option order. And generation, which it simply doesn't do. Several of these you've already met in this course.","params":{"title":"Jev 1.13's known failure modes","cols":["Failure mode","Do this instead"],"widths":"1fr 1.5fr","rows":[{"cells":["Literal reading","Write the exact condition; put boundary cases in criteria"],"at":"Literal reading"},{"cells":["Math and numbers","Keep the arithmetic in code"],"at":"Math"},{"cells":["Date comparison","Extract the parts; compare in code"],"at":"Date comparison"},{"cells":["Indirection","Reduce hops; point at the relevant state"],"at":"Indirection"},{"cells":["Large, noisy state","Filter first; send only what the question needs"],"at":"noisy state"},{"cells":["Adversarial content","Be explicit; test edge cases before deploying"],"at":"Adversarial"},{"cells":["Contradicting instructions/criteria","Align them; true should mean yes"],"at":"contradict"},{"cells":["Option order","Reorder options; check the answer holds"],"at":"Option order"},{"cells":["Generation","Use a generative model"],"hl":true,"at":"generation"}],"note":"TypeSafe docs: Jev 1.13 jaggedness."},"start":88.726,"dur":25.949,"words":[{"text":"TypeSafe","s":89.026,"e":89.586},{"text":"publishes","s":89.666,"e":90.146},{"text":"the","s":90.186,"e":90.226},{"text":"full","s":90.386,"e":90.546},{"text":"list","s":90.626,"e":90.866},{"text":"for","s":90.946,"e":91.026},{"text":"Jev","s":91.106,"e":91.346},{"text":"one","s":91.426,"e":91.506},{"text":"point","s":91.586,"e":91.746},{"text":"thirteen,","s":91.826,"e":92.786},{"text":"nine","s":92.866,"e":93.106},{"text":"failure","s":93.266,"e":93.586},{"text":"modes,","s":93.666,"e":94.226},{"text":"each","s":94.306,"e":94.546},{"text":"with","s":94.626,"e":94.706},{"text":"a","s":94.786,"e":94.826},{"text":"fix.","s":94.866,"e":95.626},{"text":"Literal","s":95.906,"e":96.226},{"text":"reading.","s":96.306,"e":96.786},{"text":"Math.","s":97.186,"e":97.746},{"text":"Date","s":98.066,"e":98.306},{"text":"comparison.","s":98.386,"e":99.106},{"text":"Indirection.","s":99.346,"e":100.226},{"text":"Large,","s":100.546,"e":101.106},{"text":"noisy","s":101.186,"e":101.586},{"text":"state.","s":101.666,"e":102.146},{"text":"Adversarial","s":102.386,"e":103.106},{"text":"content.","s":103.186,"e":103.826},{"text":"Instructions","s":104.066,"e":104.706},{"text":"that","s":104.786,"e":104.866},{"text":"contradict","s":104.946,"e":105.506},{"text":"the","s":105.586,"e":105.666},{"text":"criteria.","s":105.746,"e":106.466},{"text":"Option","s":106.786,"e":107.106},{"text":"order.","s":107.186,"e":107.586},{"text":"And","s":107.826,"e":108.066},{"text":"generation,","s":108.306,"e":109.186},{"text":"which","s":109.346,"e":109.506},{"text":"it","s":109.586,"e":109.746},{"text":"simply","s":109.826,"e":110.146},{"text":"doesn't","s":110.306,"e":110.626},{"text":"do.","s":110.706,"e":111.026},{"text":"Several","s":111.506,"e":111.746},{"text":"of","s":111.826,"e":111.906},{"text":"these","s":112.026,"e":112.146},{"text":"you've","s":112.226,"e":112.386},{"text":"already","s":112.546,"e":112.706},{"text":"met","s":112.786,"e":112.946},{"text":"in","s":112.986,"e":113.026},{"text":"this","s":113.106,"e":113.186},{"text":"course.","s":113.266,"e":114.066}]},{"template":"statement","chapter":"Wrong, and not unsure","narration":"Now the uncomfortable part. Low confidence won't always warn you. A poker player tested Jev on a spot every regular knows: you hold the best possible hand, so you check and let the other player bet. The solver checks every time. Jev went all in, sixteen runs out of sixteen.","params":{"eyebrow":"Mike Ramos, “Jev is the fish at the poker table”","text":"Best possible hand: the solver checks *100%*. Jev shoved *16 runs out of 16*.","sub":"Typed output is never malformed. It can still be wrong, without sounding unsure.","sub_at":"every time","art":"dice","tone":"bad","punch":"sixteen runs out of sixteen"},"start":114.675,"dur":20.907,"words":[{"text":"Now","s":114.975,"e":115.295},{"text":"the","s":115.375,"e":115.455},{"text":"uncomfortable","s":115.535,"e":116.175},{"text":"part.","s":116.255,"e":116.735},{"text":"Low","s":117.295,"e":117.455},{"text":"confidence","s":117.535,"e":118.175},{"text":"won't","s":118.415,"e":118.655},{"text":"always","s":118.735,"e":119.055},{"text":"warn","s":119.135,"e":119.375},{"text":"you.","s":119.455,"e":119.935},{"text":"A","s":120.175,"e":120.335},{"text":"poker","s":120.415,"e":120.575},{"text":"player","s":120.655,"e":120.895},{"text":"tested","s":120.975,"e":121.295},{"text":"Jev","s":121.375,"e":121.695},{"text":"on","s":121.735,"e":121.775},{"text":"a","s":121.855,"e":121.895},{"text":"spot","s":121.935,"e":122.255},{"text":"every","s":122.335,"e":122.655},{"text":"regular","s":122.735,"e":123.135},{"text":"knows:","s":123.215,"e":124.015},{"text":"you","s":124.095,"e":124.175},{"text":"hold","s":124.255,"e":124.415},{"text":"the","s":124.455,"e":124.495},{"text":"best","s":124.655,"e":124.895},{"text":"possible","s":124.975,"e":125.295},{"text":"hand,","s":125.375,"e":125.855},{"text":"so","s":125.935,"e":126.175},{"text":"you","s":126.255,"e":126.335},{"text":"check","s":126.415,"e":126.815},{"text":"and","s":126.895,"e":126.975},{"text":"let","s":127.055,"e":127.135},{"text":"the","s":127.215,"e":127.295},{"text":"other","s":127.375,"e":127.455},{"text":"player","s":127.615,"e":127.775},{"text":"bet.","s":127.855,"e":128.655},{"text":"The","s":128.695,"e":128.735},{"text":"solver","s":128.815,"e":129.215},{"text":"checks","s":129.375,"e":129.775},{"text":"every","s":129.855,"e":130.175},{"text":"time.","s":130.375,"e":131.055},{"text":"Jev","s":131.135,"e":131.375},{"text":"went","s":131.455,"e":131.775},{"text":"all","s":131.855,"e":132.015},{"text":"in,","s":132.095,"e":132.655},{"text":"sixteen","s":132.735,"e":133.295},{"text":"runs","s":133.535,"e":133.775},{"text":"out","s":133.815,"e":133.855},{"text":"of","s":133.935,"e":134.015},{"text":"sixteen.","s":134.095,"e":134.975}]},{"template":"quote","chapter":"Wrong, and not unsure","narration":"That's why the claim that a decision model can't hallucinate needs care. It can't return an invalid type or an option that doesn't exist. But as TypeSafe's CEO wrote on launch day, it's also possible to be confidently wrong, and all future models will still have that possibility.","params":{"text":"It's also possible to be *confidently wrong* (and all future models will be smarter still and still have that possibility).","who":"Diogo Almeida","role":"TypeSafe AI CEO, on Hacker News","src":"Introducing System One Models and Jev, launch thread"},"start":135.582,"dur":18.818,"words":[{"text":"That's","s":135.882,"e":136.175},{"text":"why","s":136.229,"e":136.282},{"text":"the","s":136.362,"e":136.442},{"text":"claim","s":136.602,"e":136.762},{"text":"that","s":136.842,"e":136.922},{"text":"a","s":136.962,"e":137.002},{"text":"decision","s":137.242,"e":137.482},{"text":"model","s":137.562,"e":137.802},{"text":"can't","s":137.962,"e":138.202},{"text":"hallucinate","s":138.282,"e":138.922},{"text":"needs","s":139.002,"e":139.242},{"text":"care.","s":139.322,"e":139.802},{"text":"It","s":140.282,"e":140.362},{"text":"can't","s":140.442,"e":140.602},{"text":"return","s":140.802,"e":141.002},{"text":"an","s":141.082,"e":141.162},{"text":"invalid","s":141.242,"e":141.642},{"text":"type","s":141.722,"e":142.122},{"text":"or","s":142.202,"e":142.362},{"text":"an","s":142.442,"e":142.522},{"text":"option","s":142.602,"e":142.842},{"text":"that","s":142.922,"e":143.002},{"text":"doesn't","s":143.082,"e":143.242},{"text":"exist.","s":143.562,"e":144.122},{"text":"But","s":144.362,"e":144.602},{"text":"as","s":144.682,"e":144.842},{"text":"TypeSafe's","s":144.922,"e":145.482},{"text":"CEO","s":145.562,"e":145.962},{"text":"wrote","s":146.042,"e":146.202},{"text":"on","s":146.282,"e":146.362},{"text":"launch","s":146.442,"e":146.682},{"text":"day,","s":146.762,"e":147.322},{"text":"it's","s":147.402,"e":147.642},{"text":"also","s":147.802,"e":147.962},{"text":"possible","s":148.042,"e":148.522},{"text":"to","s":148.602,"e":148.682},{"text":"be","s":148.722,"e":148.762},{"text":"confidently","s":148.922,"e":149.562},{"text":"wrong,","s":149.642,"e":150.442},{"text":"and","s":150.522,"e":150.682},{"text":"all","s":150.802,"e":150.922},{"text":"future","s":151.002,"e":151.242},{"text":"models","s":151.322,"e":151.722},{"text":"will","s":151.842,"e":151.962},{"text":"still","s":152.042,"e":152.282},{"text":"have","s":152.442,"e":152.602},{"text":"that","s":152.682,"e":152.762},{"text":"possibility.","s":152.842,"e":153.802}]},{"template":"list","chapter":"What not to ask","narration":"So here's what not to ask a decision model. Don't ask it to count, add, or compare numbers and dates. Don't ask it to write anything. Don't ask it to predict the real world: one tester asked Clef which thumbnails would beat a channel's average, and its picks did exactly as well as the average, a coin flip. And don't ask for long chains of reasoning in one judgment: an independent test found OpenAI's Decisions confidence no better than chance five reasoning steps deep.","params":{"title":"Don't ask a decision model to…","style":"x","items":[{"text":"**count, add, or compare** numbers and dates","at":"count, add"},{"text":"**write** anything: replies, code, arguments","at":"write anything"},{"text":"**forecast** the real world (thumbnail picks: a coin flip)","at":"predict the real world"},{"text":"chain many **reasoning steps** in one judgment","at":"long chains"}]},"start":154.4,"dur":31.304,"words":[{"text":"So","s":154.7,"e":154.86},{"text":"here's","s":154.94,"e":155.18},{"text":"what","s":155.3,"e":155.42},{"text":"not","s":155.58,"e":155.9},{"text":"to","s":156.06,"e":156.14},{"text":"ask","s":156.22,"e":156.38},{"text":"a","s":156.46,"e":156.5},{"text":"decision","s":156.54,"e":156.86},{"text":"model.","s":156.94,"e":157.34},{"text":"Don't","s":157.82,"e":158.06},{"text":"ask","s":158.18,"e":158.3},{"text":"it","s":158.34,"e":158.38},{"text":"to","s":158.46,"e":158.54},{"text":"count,","s":158.62,"e":159.26},{"text":"add,","s":159.42,"e":159.66},{"text":"or","s":159.74,"e":159.82},{"text":"compare","s":159.9,"e":160.38},{"text":"numbers","s":160.54,"e":160.86},{"text":"and","s":160.9,"e":160.94},{"text":"dates.","s":161.02,"e":161.74},{"text":"Don't","s":161.98,"e":162.22},{"text":"ask","s":162.3,"e":162.46},{"text":"it","s":162.54,"e":162.62},{"text":"to","s":162.7,"e":162.78},{"text":"write","s":162.94,"e":163.18},{"text":"anything.","s":163.46,"e":164.02},{"text":"Don't","s":164.3,"e":164.54},{"text":"ask","s":164.62,"e":164.7},{"text":"it","s":164.78,"e":164.86},{"text":"to","s":164.9,"e":164.94},{"text":"predict","s":165.02,"e":165.34},{"text":"the","s":165.42,"e":165.5},{"text":"real","s":165.58,"e":165.82},{"text":"world:","s":165.9,"e":166.7},{"text":"one","s":166.78,"e":166.94},{"text":"tester","s":167.14,"e":167.34},{"text":"asked","s":167.42,"e":167.66},{"text":"Clef","s":167.74,"e":168.06},{"text":"which","s":168.14,"e":168.38},{"text":"thumbnails","s":168.46,"e":168.94},{"text":"would","s":169.02,"e":169.1},{"text":"beat","s":169.18,"e":169.34},{"text":"a","s":169.38,"e":169.42},{"text":"channel's","s":169.5,"e":169.82},{"text":"average,","s":169.9,"e":170.7},{"text":"and","s":170.78,"e":170.86},{"text":"its","s":170.94,"e":171.02},{"text":"picks","s":171.1,"e":171.34},{"text":"did","s":171.42,"e":171.58},{"text":"exactly","s":171.66,"e":172.3},{"text":"as","s":172.38,"e":172.46},{"text":"well","s":172.54,"e":172.62},{"text":"as","s":172.66,"e":172.7},{"text":"the","s":172.78,"e":172.86},{"text":"average,","s":173.1,"e":173.66},{"text":"a","s":173.7,"e":173.74},{"text":"coin","s":173.82,"e":174.06},{"text":"flip.","s":174.14,"e":174.7},{"text":"And","s":174.94,"e":175.1},{"text":"don't","s":175.18,"e":175.34},{"text":"ask","s":175.42,"e":175.66},{"text":"for","s":175.74,"e":175.82},{"text":"long","s":175.9,"e":176.14},{"text":"chains","s":176.22,"e":176.54},{"text":"of","s":176.58,"e":176.62},{"text":"reasoning","s":176.7,"e":177.1},{"text":"in","s":177.18,"e":177.26},{"text":"one","s":177.42,"e":177.58},{"text":"judgment:","s":177.66,"e":178.62},{"text":"an","s":178.66,"e":178.7},{"text":"independent","s":178.78,"e":179.26},{"text":"test","s":179.34,"e":179.66},{"text":"found","s":179.74,"e":179.98},{"text":"OpenAI's","s":180.06,"e":180.62},{"text":"Decisions","s":180.7,"e":181.18},{"text":"confidence","s":181.26,"e":181.82},{"text":"no","s":181.98,"e":182.14},{"text":"better","s":182.22,"e":182.46},{"text":"than","s":182.54,"e":182.62},{"text":"chance","s":182.7,"e":183.18},{"text":"five","s":183.42,"e":183.66},{"text":"reasoning","s":183.74,"e":184.14},{"text":"steps","s":184.3,"e":184.46},{"text":"deep.","s":184.54,"e":185.1}]},{"template":"statement","chapter":"The rule","narration":"Which leaves one rule, best put in the README of the open model Jeff. Reason in code, decide with the model. Code does the counting, the sums, the dates and the steps. The model makes the judgment that code can't.","params":{"eyebrow":"The Jeff README","text":"Reason in code, *decide with the model*.","sub":"Code counts, adds, compares dates and chains steps. The model judges meaning.","sub_at":"Code does the counting","art":"gauge","tone":"agent","punch":"decide with the model"},"start":185.704,"dur":15.866,"words":[{"text":"Which","s":186.004,"e":186.244},{"text":"leaves","s":186.284,"e":186.564},{"text":"one","s":186.724,"e":186.884},{"text":"rule,","s":186.964,"e":187.764},{"text":"best","s":187.844,"e":188.084},{"text":"put","s":188.164,"e":188.324},{"text":"in","s":188.364,"e":188.404},{"text":"the","s":188.484,"e":188.524},{"text":"README","s":188.564,"e":189.044},{"text":"of","s":189.124,"e":189.204},{"text":"the","s":189.244,"e":189.284},{"text":"open","s":189.444,"e":189.684},{"text":"model","s":189.764,"e":190.004},{"text":"Jeff.","s":190.164,"e":190.644},{"text":"Reason","s":191.124,"e":191.444},{"text":"in","s":191.524,"e":191.604},{"text":"code,","s":191.684,"e":192.244},{"text":"decide","s":192.404,"e":192.964},{"text":"with","s":193.044,"e":193.204},{"text":"the","s":193.244,"e":193.284},{"text":"model.","s":193.364,"e":194.044},{"text":"Code","s":194.324,"e":194.564},{"text":"does","s":194.644,"e":194.724},{"text":"the","s":194.804,"e":194.844},{"text":"counting,","s":194.884,"e":195.604},{"text":"the","s":195.644,"e":195.684},{"text":"sums,","s":195.764,"e":196.324},{"text":"the","s":196.364,"e":196.404},{"text":"dates","s":196.484,"e":196.884},{"text":"and","s":196.964,"e":197.044},{"text":"the","s":197.084,"e":197.124},{"text":"steps.","s":197.204,"e":198.004},{"text":"The","s":198.084,"e":198.164},{"text":"model","s":198.244,"e":198.644},{"text":"makes","s":198.724,"e":198.884},{"text":"the","s":198.964,"e":199.044},{"text":"judgment","s":199.124,"e":199.524},{"text":"that","s":199.604,"e":199.764},{"text":"code","s":199.844,"e":200.164},{"text":"can't.","s":200.244,"e":200.964}]},{"template":"quiz","chapter":"Quick check","narration":"Quick check. You want to flag orders where the customer was charged more than the quoted price. Which design is right?","params":{"q":"Flag orders where the customer was **charged more than the quote**. Which design?","options":["A Noul: “Was the customer overcharged?”","Select both amounts from candidates, then compare them in code","A Score for how overcharged they were","A Choice between “overcharged” and “fine”, plus “unsure”"],"answer":1,"explain":"Comparing numbers is code's job. Use the model to pick which amount is the quote and which is the charge, then compare them exactly."},"start":201.57,"dur":12.657,"words":[{"text":"Quick","s":201.87,"e":202.11},{"text":"check.","s":202.19,"e":202.71},{"text":"You","s":202.91,"e":203.07},{"text":"want","s":203.15,"e":203.23},{"text":"to","s":203.31,"e":203.39},{"text":"flag","s":203.55,"e":203.71},{"text":"orders","s":203.79,"e":204.19},{"text":"where","s":204.27,"e":204.35},{"text":"the","s":204.39,"e":204.43},{"text":"customer","s":204.51,"e":204.91},{"text":"was","s":205.03,"e":205.15},{"text":"charged","s":205.35,"e":205.55},{"text":"more","s":205.71,"e":205.95},{"text":"than","s":206.03,"e":206.11},{"text":"the","s":206.19,"e":206.27},{"text":"quoted","s":206.35,"e":206.59},{"text":"price.","s":206.67,"e":207.63},{"text":"Which","s":207.75,"e":207.87},{"text":"design","s":208.03,"e":208.19},{"text":"is","s":208.27,"e":208.35},{"text":"right?","s":208.43,"e":209.07}]},{"template":"exercise","chapter":"Try it","narration":"Try it. Have your agent audit your questions for jagged edges.","params":{"title":"Audit for jagged edges","intro":"Paste this into your coding agent:","prompt":"Audit every decision-model question in this codebase against the known\nfailure modes: counting, arithmetic, number or date comparison,\nindirection (questions that point at one item in a long list), large\nirrelevant state, adversarial input, criteria that contradict the\ninstruction, option order, and anything that asks for generation. For\neach hit, rewrite it so code does the computing and the model only\njudges meaning, and add a test with a known answer for each rewrite."},"start":214.227,"dur":5.234,"words":[{"text":"Try","s":214.527,"e":214.847},{"text":"it.","s":214.927,"e":215.167},{"text":"Have","s":215.487,"e":215.647},{"text":"your","s":215.727,"e":215.807},{"text":"agent","s":215.887,"e":216.207},{"text":"audit","s":216.287,"e":216.527},{"text":"your","s":216.607,"e":216.687},{"text":"questions","s":216.767,"e":217.247},{"text":"for","s":217.327,"e":217.487},{"text":"jagged","s":217.567,"e":217.887},{"text":"edges.","s":218.047,"e":218.847}]},{"template":"recap","chapter":"Recap","narration":"So: decision models are jagged. Strong on meaning, weak on counting, arithmetic and dates. Typed output can still be wrong, sometimes confidently. Know the nine failure modes, and reason in code, decide with the model. Next: proving it works, on your own data.","params":{"items":["Jagged: great at meaning, unreliable at counting, sums, dates.","Count with one Noul per item; add up in code.","Never malformed is not never wrong.","Reason in code, decide with the model."],"next":"Evaluate on your own data","next_at":"Next"},"start":219.461,"dur":21.309,"words":[{"text":"So:","s":219.761,"e":220.161},{"text":"decision","s":220.481,"e":220.911},{"text":"models","s":220.961,"e":221.374},{"text":"are","s":221.441,"e":221.574},{"text":"jagged.","s":221.601,"e":222.241},{"text":"Strong","s":222.321,"e":222.988},{"text":"on","s":223.041,"e":223.161},{"text":"meaning,","s":223.201,"e":223.681},{"text":"weak","s":224.001,"e":224.211},{"text":"on","s":224.241,"e":224.401},{"text":"counting,","s":224.481,"e":225.201},{"text":"arithmetic","s":225.281,"e":225.921},{"text":"and","s":226.001,"e":226.081},{"text":"dates.","s":226.161,"e":226.841},{"text":"Typed","s":227.041,"e":227.361},{"text":"output","s":227.441,"e":227.841},{"text":"can","s":227.921,"e":228.081},{"text":"still","s":228.161,"e":228.321},{"text":"be","s":228.401,"e":228.481},{"text":"wrong,","s":228.561,"e":229.201},{"text":"sometimes","s":229.441,"e":229.681},{"text":"confidently.","s":229.761,"e":230.641},{"text":"Know","s":230.881,"e":231.041},{"text":"the","s":231.121,"e":231.201},{"text":"nine","s":231.361,"e":231.601},{"text":"failure","s":231.681,"e":232.001},{"text":"modes,","s":232.081,"e":232.561},{"text":"and","s":232.641,"e":232.721},{"text":"reason","s":232.801,"e":233.121},{"text":"in","s":233.161,"e":233.201},{"text":"code,","s":233.361,"e":233.921},{"text":"decide","s":234.001,"e":234.481},{"text":"with","s":234.561,"e":234.721},{"text":"the","s":234.761,"e":234.801},{"text":"model.","s":234.881,"e":235.281},{"text":"Next:","s":236.321,"e":236.801},{"text":"proving","s":236.881,"e":237.121},{"text":"it","s":237.201,"e":237.281},{"text":"works,","s":237.681,"e":238.001},{"text":"on","s":238.081,"e":238.161},{"text":"your","s":238.241,"e":238.321},{"text":"own","s":238.401,"e":238.481},{"text":"data.","s":238.881,"e":239.201}]}],"total":241.17,"audio":"/audio/19-jagged-edges.mp3"}