{"id":"22-capstone","number":22,"slug":"capstone-support-triage","title":"Capstone: a support triage pipeline","short":"Capstone","module":"Shipping it","summary":"The whole course in one pipeline you can run: code prepares the state, one request asks fifteen questions, and code routes every ticket to a refund, a reset link, an LLM, a log, a person or a block. Here is how we built it, the three bad questions our first run exposed, what held-out tickets showed, and why twelve out of twelve still proves less than it sounds.","voice":"Jessica","scenes":[{"template":"title","chapter":"Intro","narration":"Time to put it all together. One pipeline, built from everything in this course, that triages a support inbox. The code is in the course repository, and you can run it yourself.","params":{"kicker":"Lesson 22","module":"Module 6 · Shipping it","title":"Capstone: a *support triage* pipeline","subtitle":"Everything in this course, in one pipeline you can run.","cast":["snap","curly"]},"start":0.5,"dur":12.119,"words":[{"text":"Time","s":0.8,"e":1.04},{"text":"to","s":1.12,"e":1.2},{"text":"put","s":1.28,"e":1.44},{"text":"it","s":1.52,"e":1.6},{"text":"all","s":1.68,"e":1.76},{"text":"together.","s":1.84,"e":2.32},{"text":"One","s":2.8,"e":2.96},{"text":"pipeline,","s":3.04,"e":3.84},{"text":"built","s":3.96,"e":4.08},{"text":"from","s":4.16,"e":4.24},{"text":"everything","s":4.48,"e":4.96},{"text":"in","s":5,"e":5.04},{"text":"this","s":5.12,"e":5.2},{"text":"course,","s":5.28,"e":5.84},{"text":"that","s":5.92,"e":6},{"text":"triages","s":6.08,"e":6.72},{"text":"a","s":6.8,"e":6.84},{"text":"support","s":6.88,"e":7.28},{"text":"inbox.","s":7.56,"e":7.92},{"text":"The","s":8.32,"e":8.4},{"text":"code","s":8.48,"e":8.72},{"text":"is","s":8.76,"e":8.8},{"text":"in","s":8.88,"e":8.92},{"text":"the","s":8.96,"e":9},{"text":"course","s":9.04,"e":9.28},{"text":"repository,","s":9.64,"e":10.16},{"text":"and","s":10.28,"e":10.4},{"text":"you","s":10.48,"e":10.56},{"text":"can","s":10.64,"e":10.72},{"text":"run","s":10.76,"e":10.8},{"text":"it","s":10.88,"e":10.96},{"text":"yourself.","s":11.04,"e":11.76}]},{"template":"flow","chapter":"The design","narration":"Here's the design. First, code prepares the state. It redacts card numbers, so no model ever sees one, and lists the customer's orders as candidates. Then one request asks fifteen questions at once, in a quarter of a second. Then code decides: thresholds, policy checks, and a log of every answer.","params":{"title":"Code prepares, the model judges, code decides","nodes":[{"label":"Code prepares","sub":"redact cards · candidate orders · company info","kind":"curly","edge":"state","at":"code prepares"},{"label":"One request","sub":"15 questions · ~0.25 s","kind":"snap","edge":"answers","at":"one request"},{"label":"Code decides","sub":"thresholds · policy checks · log","kind":"curly","at":"code decides"}]},"start":12.619,"dur":22.501,"words":[{"text":"Here's","s":12.919,"e":13.199},{"text":"the","s":13.239,"e":13.319},{"text":"design.","s":13.559,"e":13.879},{"text":"First,","s":14.279,"e":14.999},{"text":"code","s":15.159,"e":15.319},{"text":"prepares","s":15.399,"e":15.879},{"text":"the","s":15.959,"e":16.039},{"text":"state.","s":16.119,"e":16.599},{"text":"It","s":16.999,"e":17.079},{"text":"redacts","s":17.199,"e":17.559},{"text":"card","s":17.639,"e":17.879},{"text":"numbers,","s":17.959,"e":18.439},{"text":"so","s":18.519,"e":18.679},{"text":"no","s":18.839,"e":18.999},{"text":"model","s":19.079,"e":19.319},{"text":"ever","s":19.479,"e":19.719},{"text":"sees","s":19.879,"e":20.119},{"text":"one,","s":20.239,"e":20.679},{"text":"and","s":20.759,"e":20.999},{"text":"lists","s":21.159,"e":21.399},{"text":"the","s":21.439,"e":21.479},{"text":"customer's","s":21.559,"e":22.039},{"text":"orders","s":22.239,"e":22.439},{"text":"as","s":22.519,"e":22.599},{"text":"candidates.","s":22.679,"e":23.319},{"text":"Then","s":23.799,"e":24.119},{"text":"one","s":24.439,"e":24.599},{"text":"request","s":24.839,"e":25.079},{"text":"asks","s":25.159,"e":25.479},{"text":"fifteen","s":25.639,"e":26.119},{"text":"questions","s":26.199,"e":26.599},{"text":"at","s":26.639,"e":26.679},{"text":"once,","s":26.839,"e":27.239},{"text":"in","s":27.279,"e":27.319},{"text":"a","s":27.399,"e":27.479},{"text":"quarter","s":27.559,"e":27.879},{"text":"of","s":27.959,"e":28.039},{"text":"a","s":28.092,"e":28.146},{"text":"second.","s":28.199,"e":28.679},{"text":"Then","s":29.159,"e":29.479},{"text":"code","s":29.639,"e":29.799},{"text":"decides:","s":29.879,"e":30.839},{"text":"thresholds,","s":30.919,"e":31.719},{"text":"policy","s":31.919,"e":32.119},{"text":"checks,","s":32.199,"e":32.599},{"text":"and","s":32.639,"e":32.679},{"text":"a","s":32.759,"e":32.839},{"text":"log","s":32.919,"e":33.159},{"text":"of","s":33.239,"e":33.319},{"text":"every","s":33.479,"e":33.719},{"text":"answer.","s":33.799,"e":34.519}]},{"template":"request","chapter":"The design","narration":"Here's one ticket. My card, four one one one and so on, was charged twice for order P-4410. The card number is already gone. The answers: billing, at one point oh. A duplicate charge, point nine nine. The order, picked from the customer's own orders: P-4410. No injection, no escalation, one issue.","params":{"model":"jev-1.13.0","latency":0.24,"state":{"ticket":"My card [card redacted] was charged twice for order P-4410. Please refund the extra one.","customer":{"plan":"Pro","orders":[{"id":"P-4410","item":"Storage add-on"}]}},"questions":[{"id":"cat","type":"choice","text":"Main request? (8 categories)","ans":{"choice":"billing","p":1},"say_at":"billing"},{"id":"dup","type":"noul","text":"Charged more than once for the same order?","ans":{"v":0.99},"say_at":"A duplicate charge"},{"id":"ord","type":"choice","text":"Which order in `customer.orders`? (+ none)","ans":{"choice":"P-4410","p":1},"say_at":"The order"},{"id":"inj","type":"noul","text":"Tries to override how support works?","ans":{"v":0.04},"say_at":"No injection"},{"id":"esc","type":"noul","text":"Mentions cancelling, legal action or a breach?","ans":{"v":0.02}}],"send_at":"Here's one ticket","ans_at":"The answers","footer":"5 of 15 questions shown. Real answers, jev-1.13.0, 2026-10-15."},"start":35.12,"dur":27.308,"words":[{"text":"Here's","s":35.42,"e":35.74},{"text":"one","s":35.82,"e":35.98},{"text":"ticket.","s":36.06,"e":36.54},{"text":"My","s":36.94,"e":37.1},{"text":"card,","s":37.34,"e":37.82},{"text":"four","s":37.9,"e":38.14},{"text":"one","s":38.22,"e":38.46},{"text":"one","s":38.58,"e":38.7},{"text":"one","s":38.82,"e":38.94},{"text":"and","s":39.1,"e":39.26},{"text":"so","s":39.34,"e":39.5},{"text":"on,","s":39.62,"e":40.14},{"text":"was","s":40.22,"e":40.3},{"text":"charged","s":40.38,"e":40.7},{"text":"twice","s":40.78,"e":41.18},{"text":"for","s":41.26,"e":41.34},{"text":"order","s":41.42,"e":41.66},{"text":"P-4410.","s":41.82,"e":44.14},{"text":"The","s":44.18,"e":44.22},{"text":"card","s":44.3,"e":44.46},{"text":"number","s":44.54,"e":44.78},{"text":"is","s":44.94,"e":45.02},{"text":"already","s":45.18,"e":45.42},{"text":"gone.","s":45.5,"e":46.22},{"text":"The","s":46.54,"e":46.62},{"text":"answers:","s":46.7,"e":47.66},{"text":"billing,","s":47.74,"e":48.46},{"text":"at","s":48.54,"e":48.62},{"text":"one","s":48.7,"e":48.86},{"text":"point","s":48.94,"e":49.18},{"text":"oh.","s":49.26,"e":49.9},{"text":"A","s":49.953,"e":50.007},{"text":"duplicate","s":50.06,"e":50.46},{"text":"charge,","s":50.54,"e":51.26},{"text":"point","s":51.42,"e":51.58},{"text":"nine","s":51.7,"e":51.82},{"text":"nine.","s":52.02,"e":52.7},{"text":"The","s":52.74,"e":52.78},{"text":"order,","s":52.94,"e":53.58},{"text":"picked","s":53.7,"e":53.82},{"text":"from","s":53.9,"e":53.98},{"text":"the","s":54.02,"e":54.06},{"text":"customer's","s":54.14,"e":54.7},{"text":"own","s":54.82,"e":54.94},{"text":"orders:","s":55.02,"e":55.82},{"text":"P-4410.","s":55.98,"e":58.14},{"text":"No","s":58.22,"e":58.38},{"text":"injection,","s":58.46,"e":59.34},{"text":"no","s":59.42,"e":59.5},{"text":"escalation,","s":59.58,"e":60.7},{"text":"one","s":60.78,"e":60.94},{"text":"issue.","s":61.02,"e":61.82}]},{"template":"code","chapter":"Meaning and facts","narration":"Now the refund rule. The model must hear a duplicate charge, at point eight or more, the threshold we derived from costs in lesson ten. It must be sure which order. And then code checks the facts: the records really show two identical charges, and the amount is under the hundred dollar limit in the policy. The model judges meaning. Code checks facts. Both have to agree before money moves.","params":{"file":"triage.py","lang":"py","lines":["if cat == \"billing\" and n(\"reports_duplicate\") >= ACT:   # 0.8","    order = find(customer.orders, a[\"order\"].choice)","    if order is None or a[\"order\"].confidence < 0.9:","        return \"human\", \"can't tell which order\"","    if not has_identical_charges(order):","        return \"human\", \"claimed duplicate not in our records\"","    if order.charges[0] > CREDIT_LIMIT_USD:   # policy: $100","        return \"human\", \"over the credit limit\"","    return \"auto_refund\", \"duplicate confirmed in records\""],"hl":[{"from":1,"to":3,"label":"the model: meaning","tone":"agent","at":"must hear a duplicate"},{"from":5,"to":8,"label":"code: facts","tone":"good","at":"code checks the facts"}]},"start":62.428,"dur":27.464,"words":[{"text":"Now","s":62.728,"e":63.048},{"text":"the","s":63.128,"e":63.208},{"text":"refund","s":63.288,"e":63.688},{"text":"rule.","s":63.768,"e":64.168},{"text":"The","s":64.648,"e":64.728},{"text":"model","s":64.808,"e":65.048},{"text":"must","s":65.208,"e":65.448},{"text":"hear","s":65.528,"e":65.688},{"text":"a","s":65.728,"e":65.768},{"text":"duplicate","s":65.848,"e":66.248},{"text":"charge,","s":66.328,"e":66.888},{"text":"at","s":66.968,"e":67.048},{"text":"point","s":67.128,"e":67.368},{"text":"eight","s":67.528,"e":67.608},{"text":"or","s":67.768,"e":67.848},{"text":"more,","s":67.928,"e":68.568},{"text":"the","s":68.608,"e":68.648},{"text":"threshold","s":68.728,"e":69.128},{"text":"we","s":69.208,"e":69.288},{"text":"derived","s":69.488,"e":69.688},{"text":"from","s":69.768,"e":69.848},{"text":"costs","s":69.928,"e":70.288},{"text":"in","s":70.328,"e":70.408},{"text":"lesson","s":70.568,"e":70.728},{"text":"ten.","s":70.808,"e":71.208},{"text":"It","s":71.688,"e":71.768},{"text":"must","s":71.928,"e":72.088},{"text":"be","s":72.168,"e":72.248},{"text":"sure","s":72.328,"e":72.568},{"text":"which","s":72.808,"e":73.048},{"text":"order.","s":73.128,"e":73.608},{"text":"And","s":74.088,"e":74.248},{"text":"then","s":74.328,"e":74.568},{"text":"code","s":74.808,"e":75.048},{"text":"checks","s":75.128,"e":75.368},{"text":"the","s":75.448,"e":75.488},{"text":"facts:","s":75.528,"e":76.408},{"text":"the","s":76.448,"e":76.488},{"text":"records","s":76.568,"e":76.968},{"text":"really","s":77.128,"e":77.448},{"text":"show","s":77.528,"e":77.768},{"text":"two","s":77.928,"e":78.088},{"text":"identical","s":78.168,"e":78.568},{"text":"charges,","s":78.728,"e":79.528},{"text":"and","s":79.648,"e":79.768},{"text":"the","s":79.808,"e":79.848},{"text":"amount","s":80.048,"e":80.248},{"text":"is","s":80.368,"e":80.488},{"text":"under","s":80.608,"e":80.728},{"text":"the","s":80.808,"e":80.888},{"text":"hundred","s":80.928,"e":81.128},{"text":"dollar","s":81.208,"e":81.448},{"text":"limit","s":81.528,"e":81.768},{"text":"in","s":81.808,"e":81.848},{"text":"the","s":81.928,"e":81.968},{"text":"policy.","s":82.008,"e":82.848},{"text":"The","s":83.128,"e":83.208},{"text":"model","s":83.288,"e":83.528},{"text":"judges","s":83.688,"e":84.008},{"text":"meaning.","s":84.088,"e":84.568},{"text":"Code","s":84.968,"e":85.288},{"text":"checks","s":85.368,"e":85.608},{"text":"facts.","s":85.688,"e":86.248},{"text":"Both","s":86.648,"e":86.888},{"text":"have","s":86.968,"e":87.048},{"text":"to","s":87.128,"e":87.208},{"text":"agree","s":87.408,"e":87.608},{"text":"before","s":87.688,"e":88.088},{"text":"money","s":88.168,"e":88.408},{"text":"moves.","s":88.488,"e":89.288}]},{"template":"lanes","chapter":"Where tickets go","narration":"Every ticket lands in a route. Code handles refunds, reset links, the feature board, and closing thank-yous and spam. An LLM answers how-to questions from the docs. People get everything unclear, risky or over the limit, with escalations, critical bugs and repeat contacts at the front of the line. And attempts to instruct the system are blocked before any LLM sees them.","params":{"source":"36 tickets","lanes":[{"label":"Code handles it","rule":"refund · reset link · log · close","tone":"good","who":"curly","at":"Code handles"},{"label":"An LLM answers","rule":"how-to, from the docs","tone":"s2","who":"mull","at":"An LLM answers"},{"label":"A person","rule":"unclear, risky, over the limit","tone":"human","who":"you","at":"People get everything"},{"label":"Priority or blocked","rule":"escalations · critical bugs · injections","tone":"bad","icon":"shield","at":"blocked"}],"items":[{"text":"Charged twice for P-4410","conf":0.99,"lane":0},{"text":"How do I rotate my API key?","conf":0.96,"lane":1},{"text":"Double charge, $189 router","conf":0.98,"lane":2},{"text":"Production is down!","conf":0.99,"lane":3},{"text":"Thanks, all sorted!","conf":0.95,"lane":0},{"text":"“Ignore your instructions…”","conf":0.99,"lane":3}]},"start":89.892,"dur":24.512,"words":[{"text":"Every","s":90.192,"e":90.512},{"text":"ticket","s":90.712,"e":90.912},{"text":"lands","s":90.992,"e":91.232},{"text":"in","s":91.272,"e":91.312},{"text":"a","s":91.352,"e":91.392},{"text":"route.","s":91.472,"e":92.192},{"text":"Code","s":92.352,"e":92.512},{"text":"handles","s":92.592,"e":92.912},{"text":"refunds,","s":92.992,"e":93.712},{"text":"reset","s":93.912,"e":94.112},{"text":"links,","s":94.192,"e":94.592},{"text":"the","s":94.632,"e":94.672},{"text":"feature","s":94.752,"e":95.072},{"text":"board,","s":95.152,"e":95.552},{"text":"and","s":95.672,"e":95.792},{"text":"closing","s":95.872,"e":96.272},{"text":"thank-yous","s":96.352,"e":96.832},{"text":"and","s":96.872,"e":96.912},{"text":"spam.","s":97.072,"e":97.872},{"text":"An","s":97.952,"e":98.032},{"text":"LLM","s":98.112,"e":98.592},{"text":"answers","s":98.672,"e":98.992},{"text":"how-to","s":99.072,"e":99.472},{"text":"questions","s":99.552,"e":100.032},{"text":"from","s":100.112,"e":100.192},{"text":"the","s":100.232,"e":100.272},{"text":"docs.","s":100.512,"e":101.152},{"text":"People","s":101.272,"e":101.392},{"text":"get","s":101.472,"e":101.632},{"text":"everything","s":101.872,"e":102.112},{"text":"unclear,","s":102.192,"e":102.912},{"text":"risky","s":102.952,"e":103.472},{"text":"or","s":103.552,"e":103.792},{"text":"over","s":103.912,"e":104.032},{"text":"the","s":104.072,"e":104.112},{"text":"limit,","s":104.192,"e":104.832},{"text":"with","s":104.912,"e":104.992},{"text":"escalations,","s":105.072,"e":106.032},{"text":"critical","s":106.232,"e":106.432},{"text":"bugs","s":106.632,"e":106.832},{"text":"and","s":106.912,"e":106.992},{"text":"repeat","s":107.072,"e":107.392},{"text":"contacts","s":107.472,"e":107.952},{"text":"at","s":108.032,"e":108.112},{"text":"the","s":108.192,"e":108.272},{"text":"front","s":108.352,"e":108.432},{"text":"of","s":108.472,"e":108.512},{"text":"the","s":108.592,"e":108.672},{"text":"line.","s":108.832,"e":109.392},{"text":"And","s":109.432,"e":109.472},{"text":"attempts","s":109.552,"e":109.872},{"text":"to","s":109.952,"e":110.032},{"text":"instruct","s":110.112,"e":110.432},{"text":"the","s":110.472,"e":110.512},{"text":"system","s":110.592,"e":110.992},{"text":"are","s":111.072,"e":111.232},{"text":"blocked","s":111.432,"e":111.632},{"text":"before","s":111.712,"e":112.112},{"text":"any","s":112.232,"e":112.352},{"text":"LLM","s":112.512,"e":112.912},{"text":"sees","s":112.992,"e":113.232},{"text":"them.","s":113.312,"e":113.792}]},{"template":"chart","chapter":"The first run","narration":"Our first run was humbling: fourteen of twenty four tickets routed the way we'd labeled them. Error analysis found three bad questions behind every miss. We fixed them, and got twenty four of twenty four. But we'd fixed them while looking at those same tickets, so before rerunning, we wrote twelve new ones. Those came out twelve of twelve.","params":{"title":"Routes matching our labels","kind":"bar","ymax":1.1,"data":[{"x":"first run (24)","y":0.583,"label":"14/24","tone":"bad","at":"fourteen of twenty four"},{"x":"after fixing 3 questions (same 24)","y":1,"label":"24/24","tone":"s2","at":"twenty four of twenty four"},{"x":"held-out tickets (12)","y":1,"label":"12/12","hl":true,"at":"twelve of twelve"}],"src":"Tickets and labels written by us. jev-1.13.0, 2026-10-15."},"start":114.404,"dur":22.841,"words":[{"text":"Our","s":114.704,"e":114.944},{"text":"first","s":115.024,"e":115.264},{"text":"run","s":115.344,"e":115.664},{"text":"was","s":115.744,"e":115.904},{"text":"humbling:","s":115.984,"e":117.024},{"text":"fourteen","s":117.104,"e":117.664},{"text":"of","s":117.744,"e":117.904},{"text":"twenty","s":118.064,"e":118.224},{"text":"four","s":118.304,"e":118.464},{"text":"tickets","s":118.544,"e":118.944},{"text":"routed","s":118.984,"e":119.264},{"text":"the","s":119.304,"e":119.344},{"text":"way","s":119.424,"e":119.504},{"text":"we'd","s":119.584,"e":119.744},{"text":"labeled","s":119.824,"e":120.144},{"text":"them.","s":120.264,"e":120.864},{"text":"Error","s":120.984,"e":121.104},{"text":"analysis","s":121.184,"e":121.744},{"text":"found","s":121.824,"e":122.144},{"text":"three","s":122.304,"e":122.464},{"text":"bad","s":122.544,"e":122.704},{"text":"questions","s":122.864,"e":123.264},{"text":"behind","s":123.344,"e":123.744},{"text":"every","s":123.824,"e":124.064},{"text":"miss.","s":124.144,"e":124.944},{"text":"We","s":125.024,"e":125.104},{"text":"fixed","s":125.224,"e":125.344},{"text":"them,","s":125.424,"e":125.904},{"text":"and","s":125.984,"e":126.064},{"text":"got","s":126.144,"e":126.384},{"text":"twenty","s":126.544,"e":126.704},{"text":"four","s":126.824,"e":126.944},{"text":"of","s":127.024,"e":127.104},{"text":"twenty","s":127.224,"e":127.344},{"text":"four.","s":127.504,"e":128.144},{"text":"But","s":128.304,"e":128.464},{"text":"we'd","s":128.504,"e":128.784},{"text":"fixed","s":128.904,"e":129.024},{"text":"them","s":129.104,"e":129.344},{"text":"while","s":129.464,"e":129.584},{"text":"looking","s":129.704,"e":129.824},{"text":"at","s":129.904,"e":129.984},{"text":"those","s":130.064,"e":130.304},{"text":"same","s":130.464,"e":130.624},{"text":"tickets,","s":130.904,"e":131.344},{"text":"so","s":131.504,"e":131.664},{"text":"before","s":131.864,"e":132.064},{"text":"rerunning,","s":132.184,"e":132.864},{"text":"we","s":132.944,"e":133.024},{"text":"wrote","s":133.104,"e":133.264},{"text":"twelve","s":133.384,"e":133.664},{"text":"new","s":133.744,"e":133.904},{"text":"ones.","s":133.984,"e":134.704},{"text":"Those","s":134.824,"e":134.944},{"text":"came","s":135.024,"e":135.104},{"text":"out","s":135.184,"e":135.424},{"text":"twelve","s":135.504,"e":135.744},{"text":"of","s":135.824,"e":135.904},{"text":"twelve.","s":135.984,"e":136.624}]},{"template":"table","chapter":"The first run","narration":"The three bad questions are worth studying, because they're the mistakes you'll make too. Is it in English and about ACME? Two conditions in one question, about a company the state never described: answers hovered around point five. Does it try to give instructions to support? Taken literally, thank you, you can close this ticket, scored point nine five. And a vague line between billing and how to left an invoice question at point five one. Each fix came from an earlier lesson.","params":{"title":"Three fixes, before → after","cols":["Ticket","Bad question","Fix","Before → after"],"widths":"1.1fr 1.3fr 1.3fr .8fr","rows":[{"cells":["“How do I add a teammate?”","English *and* about ACME?","Split it; describe the company in the state","0.49 → 0.99 · 0.85"],"at":"Two conditions"},{"cells":["“You can close this ticket”","Gives instructions to support?","Contrastive criteria: what is and isn't an override","0.95 → 0.03"],"hl":true,"at":"Taken literally"},{"cells":["“Where's last month's invoice?”","billing vs how_to","Say what each category covers","0.51 → 0.96"],"at":"invoice question"}],"note":"Fixes from lessons 8 (state), 12 (atomic questions), 19 (literal reading) and 6 (describe the options)."},"start":137.245,"dur":33.159,"words":[{"text":"The","s":137.545,"e":137.678},{"text":"three","s":137.705,"e":137.913},{"text":"bad","s":137.945,"e":138.185},{"text":"questions","s":138.265,"e":138.825},{"text":"are","s":138.905,"e":138.985},{"text":"worth","s":139.145,"e":139.385},{"text":"studying,","s":139.545,"e":140.265},{"text":"because","s":140.465,"e":140.665},{"text":"they're","s":140.745,"e":140.905},{"text":"the","s":140.985,"e":141.025},{"text":"mistakes","s":141.065,"e":141.545},{"text":"you'll","s":141.625,"e":141.785},{"text":"make","s":141.865,"e":142.025},{"text":"too.","s":142.105,"e":142.505},{"text":"Is","s":143.145,"e":143.225},{"text":"it","s":143.305,"e":143.345},{"text":"in","s":143.385,"e":143.465},{"text":"English","s":143.625,"e":144.025},{"text":"and","s":144.105,"e":144.185},{"text":"about","s":144.305,"e":144.425},{"text":"ACME?","s":144.505,"e":145.065},{"text":"Two","s":145.465,"e":145.625},{"text":"conditions","s":145.705,"e":146.185},{"text":"in","s":146.265,"e":146.425},{"text":"one","s":146.505,"e":146.665},{"text":"question,","s":146.745,"e":147.545},{"text":"about","s":147.665,"e":147.785},{"text":"a","s":147.825,"e":147.865},{"text":"company","s":147.945,"e":148.345},{"text":"the","s":148.385,"e":148.425},{"text":"state","s":148.505,"e":148.825},{"text":"never","s":148.985,"e":149.225},{"text":"described:","s":149.305,"e":150.345},{"text":"answers","s":150.425,"e":150.825},{"text":"hovered","s":150.905,"e":151.145},{"text":"around","s":151.225,"e":151.545},{"text":"point","s":151.705,"e":151.865},{"text":"five.","s":152.025,"e":152.785},{"text":"Does","s":153.065,"e":153.145},{"text":"it","s":153.225,"e":153.305},{"text":"try","s":153.385,"e":153.465},{"text":"to","s":153.545,"e":153.625},{"text":"give","s":153.705,"e":153.785},{"text":"instructions","s":154.185,"e":154.585},{"text":"to","s":154.625,"e":154.665},{"text":"support?","s":154.745,"e":155.385},{"text":"Taken","s":155.785,"e":156.025},{"text":"literally,","s":156.105,"e":156.905},{"text":"thank","s":156.985,"e":157.145},{"text":"you,","s":157.225,"e":157.625},{"text":"you","s":157.665,"e":157.705},{"text":"can","s":157.785,"e":157.865},{"text":"close","s":157.945,"e":158.105},{"text":"this","s":158.185,"e":158.265},{"text":"ticket,","s":158.345,"e":158.905},{"text":"scored","s":158.985,"e":159.385},{"text":"point","s":159.505,"e":159.625},{"text":"nine","s":159.705,"e":159.865},{"text":"five.","s":160.025,"e":160.745},{"text":"And","s":160.985,"e":161.225},{"text":"a","s":161.385,"e":161.465},{"text":"vague","s":161.545,"e":161.865},{"text":"line","s":161.945,"e":162.185},{"text":"between","s":162.345,"e":162.505},{"text":"billing","s":162.665,"e":163.145},{"text":"and","s":163.225,"e":163.305},{"text":"how","s":163.385,"e":163.625},{"text":"to","s":163.705,"e":164.025},{"text":"left","s":164.265,"e":164.425},{"text":"an","s":164.505,"e":164.585},{"text":"invoice","s":164.665,"e":165.065},{"text":"question","s":165.145,"e":165.545},{"text":"at","s":165.625,"e":165.705},{"text":"point","s":165.865,"e":166.105},{"text":"five","s":166.185,"e":166.345},{"text":"one.","s":166.505,"e":167.305},{"text":"Each","s":167.425,"e":167.545},{"text":"fix","s":167.625,"e":167.945},{"text":"came","s":168.025,"e":168.265},{"text":"from","s":168.345,"e":168.425},{"text":"an","s":168.505,"e":168.585},{"text":"earlier","s":168.665,"e":168.985},{"text":"lesson.","s":169.225,"e":169.785}]},{"template":"statement","chapter":"What it proves","narration":"Now apply lesson twenty to our own result. Twelve out of twelve sounds perfect. But with zero errors in twelve tries, the rule of three says the true error rate could still be as high as one in four. These are thirty six tickets we wrote ourselves. Before this pipeline touches a real inbox, it needs a few hundred real tickets, labeled by people who know the policy.","params":{"eyebrow":"The rule of three","text":"0 errors in 12 still allows a true error rate of *up to 25%*.","sub":"95% upper bound ≈ 3 / n. Our 36 tickets are a smoke test, not an eval.","sub_at":"thirty six tickets","art":"gauge","tone":"s2","punch":"one in four"},"start":170.404,"dur":23.938,"words":[{"text":"Now","s":170.704,"e":170.944},{"text":"apply","s":171.104,"e":171.264},{"text":"lesson","s":171.344,"e":171.584},{"text":"twenty","s":171.744,"e":172.064},{"text":"to","s":172.144,"e":172.224},{"text":"our","s":172.304,"e":172.464},{"text":"own","s":172.544,"e":172.624},{"text":"result.","s":172.704,"e":173.664},{"text":"Twelve","s":173.744,"e":173.904},{"text":"out","s":173.984,"e":174.064},{"text":"of","s":174.144,"e":174.224},{"text":"twelve","s":174.304,"e":174.624},{"text":"sounds","s":174.704,"e":175.104},{"text":"perfect.","s":175.424,"e":176.144},{"text":"But","s":176.224,"e":176.304},{"text":"with","s":176.424,"e":176.544},{"text":"zero","s":176.624,"e":176.864},{"text":"errors","s":177.024,"e":177.344},{"text":"in","s":177.424,"e":177.584},{"text":"twelve","s":177.664,"e":177.824},{"text":"tries,","s":177.984,"e":178.704},{"text":"the","s":178.784,"e":178.824},{"text":"rule","s":178.864,"e":179.104},{"text":"of","s":179.184,"e":179.264},{"text":"three","s":179.344,"e":179.664},{"text":"says","s":179.784,"e":179.904},{"text":"the","s":179.984,"e":180.144},{"text":"true","s":180.304,"e":180.464},{"text":"error","s":180.544,"e":180.784},{"text":"rate","s":180.864,"e":181.104},{"text":"could","s":181.224,"e":181.344},{"text":"still","s":181.424,"e":181.664},{"text":"be","s":181.704,"e":181.744},{"text":"as","s":181.824,"e":181.904},{"text":"high","s":181.984,"e":182.144},{"text":"as","s":182.264,"e":182.384},{"text":"one","s":182.464,"e":182.544},{"text":"in","s":182.624,"e":182.864},{"text":"four.","s":183.064,"e":183.664},{"text":"These","s":183.744,"e":183.824},{"text":"are","s":183.904,"e":184.144},{"text":"thirty","s":184.184,"e":184.384},{"text":"six","s":184.544,"e":184.704},{"text":"tickets","s":184.784,"e":185.104},{"text":"we","s":185.184,"e":185.264},{"text":"wrote","s":185.304,"e":185.424},{"text":"ourselves.","s":185.504,"e":186.704},{"text":"Before","s":186.864,"e":187.024},{"text":"this","s":187.104,"e":187.184},{"text":"pipeline","s":187.264,"e":187.664},{"text":"touches","s":187.824,"e":187.984},{"text":"a","s":188.064,"e":188.144},{"text":"real","s":188.224,"e":188.464},{"text":"inbox,","s":188.544,"e":189.184},{"text":"it","s":189.264,"e":189.344},{"text":"needs","s":189.424,"e":189.504},{"text":"a","s":189.584,"e":189.664},{"text":"few","s":189.744,"e":189.984},{"text":"hundred","s":190.064,"e":190.464},{"text":"real","s":190.544,"e":190.704},{"text":"tickets,","s":190.784,"e":191.424},{"text":"labeled","s":191.464,"e":191.824},{"text":"by","s":191.904,"e":191.984},{"text":"people","s":192.144,"e":192.304},{"text":"who","s":192.424,"e":192.544},{"text":"know","s":192.624,"e":192.704},{"text":"the","s":192.784,"e":192.864},{"text":"policy.","s":193.144,"e":193.744}]},{"template":"stat","chapter":"What it costs","narration":"And what does it cost? Fifteen questions per ticket, about a thousand input tokens, a quarter of a second. That's about four cents per thousand tickets on Jev. Sixteen of our thirty six tickets needed no person at all, and every decision left a complete record behind.","params":{"items":[{"value":0.25,"decimals":2,"suffix":" s","label":"per ticket, 15 questions","tone":"agent","at":"a quarter of a second"},{"value":0.04,"decimals":2,"prefix":"$","label":"per 1,000 tickets (≈1,050 tokens each)","tone":"good","at":"four cents","hero":true},{"value":16,"suffix":" of 36","label":"handled without a person","tone":"s2","at":"Sixteen of our thirty six"}],"src":"jev-1.13.0 at $0.042 per million input tokens; median latency from Brazil, 2026-10-15.","punch":false},"start":194.342,"dur":18.191,"words":[{"text":"And","s":194.642,"e":194.802},{"text":"what","s":194.962,"e":195.122},{"text":"does","s":195.162,"e":195.202},{"text":"it","s":195.282,"e":195.362},{"text":"cost?","s":195.442,"e":196.002},{"text":"Fifteen","s":196.402,"e":196.722},{"text":"questions","s":196.882,"e":197.282},{"text":"per","s":197.322,"e":197.362},{"text":"ticket,","s":197.442,"e":198.082},{"text":"about","s":198.202,"e":198.322},{"text":"a","s":198.402,"e":198.482},{"text":"thousand","s":198.562,"e":198.882},{"text":"input","s":198.962,"e":199.202},{"text":"tokens,","s":199.282,"e":200.082},{"text":"a","s":200.122,"e":200.162},{"text":"quarter","s":200.242,"e":200.482},{"text":"of","s":200.562,"e":200.642},{"text":"a","s":200.695,"e":200.749},{"text":"second.","s":200.802,"e":201.602},{"text":"That's","s":201.682,"e":201.802},{"text":"about","s":201.842,"e":202.162},{"text":"four","s":202.242,"e":202.402},{"text":"cents","s":202.562,"e":202.802},{"text":"per","s":202.882,"e":202.962},{"text":"thousand","s":203.122,"e":203.442},{"text":"tickets","s":203.522,"e":203.842},{"text":"on","s":203.922,"e":204.002},{"text":"Jev.","s":204.082,"e":204.882},{"text":"Sixteen","s":205.042,"e":205.442},{"text":"of","s":205.522,"e":205.602},{"text":"our","s":205.722,"e":205.842},{"text":"thirty","s":205.922,"e":206.082},{"text":"six","s":206.242,"e":206.402},{"text":"tickets","s":206.482,"e":206.882},{"text":"needed","s":206.962,"e":207.282},{"text":"no","s":207.362,"e":207.522},{"text":"person","s":207.602,"e":207.922},{"text":"at","s":208.002,"e":208.082},{"text":"all,","s":208.162,"e":208.722},{"text":"and","s":208.842,"e":208.962},{"text":"every","s":209.042,"e":209.282},{"text":"decision","s":209.562,"e":209.842},{"text":"left","s":209.922,"e":210.082},{"text":"a","s":210.122,"e":210.162},{"text":"complete","s":210.242,"e":210.802},{"text":"record","s":210.882,"e":211.202},{"text":"behind.","s":211.282,"e":211.922}]},{"template":"list","chapter":"The whole course","narration":"Look at what went into it. A state with only what the questions need. Typed questions, each about one thing. Fifteen of them in one request. Candidates selected, never generated. Thresholds from costs. Guards in code and in questions. Routes to code, an LLM or a person. A log of every judgment. And an honest eval, held out before the fix.","params":{"title":"The course, in one pipeline","style":"num","items":[{"text":"**State** with only what the questions need","at":"A state"},{"text":"**Typed, atomic questions**, all in one request","at":"Typed questions"},{"text":"**Select** candidates; never generate values","at":"Candidates selected"},{"text":"**Thresholds** from costs; facts checked in code","at":"Thresholds from costs"},{"text":"**Route** to code, an LLM, or a person; **log** it all","at":"Routes to code"},{"text":"**Evaluate** on held-out cases, then real ones","at":"honest eval"}]},"start":212.533,"dur":27.072,"words":[{"text":"Look","s":212.833,"e":213.073},{"text":"at","s":213.113,"e":213.153},{"text":"what","s":213.233,"e":213.393},{"text":"went","s":213.473,"e":213.553},{"text":"into","s":213.633,"e":213.873},{"text":"it.","s":213.953,"e":214.393},{"text":"A","s":214.593,"e":214.673},{"text":"state","s":214.753,"e":215.073},{"text":"with","s":215.193,"e":215.313},{"text":"only","s":215.393,"e":215.713},{"text":"what","s":215.793,"e":215.873},{"text":"the","s":215.913,"e":215.953},{"text":"questions","s":216.033,"e":216.513},{"text":"need.","s":216.593,"e":217.153},{"text":"Typed","s":217.393,"e":217.633},{"text":"questions,","s":217.793,"e":218.593},{"text":"each","s":218.753,"e":218.913},{"text":"about","s":218.993,"e":219.233},{"text":"one","s":219.393,"e":219.553},{"text":"thing.","s":219.633,"e":220.033},{"text":"Fifteen","s":220.433,"e":220.913},{"text":"of","s":220.993,"e":221.073},{"text":"them","s":221.153,"e":221.313},{"text":"in","s":221.393,"e":221.553},{"text":"one","s":221.633,"e":221.793},{"text":"request.","s":221.873,"e":222.513},{"text":"Candidates","s":222.913,"e":223.393},{"text":"selected,","s":223.713,"e":224.353},{"text":"never","s":224.433,"e":224.673},{"text":"generated.","s":224.753,"e":225.393},{"text":"Thresholds","s":225.713,"e":226.353},{"text":"from","s":226.433,"e":226.593},{"text":"costs.","s":226.673,"e":227.313},{"text":"Guards","s":227.633,"e":228.033},{"text":"in","s":228.073,"e":228.113},{"text":"code","s":228.273,"e":228.673},{"text":"and","s":228.833,"e":229.073},{"text":"in","s":229.153,"e":229.233},{"text":"questions.","s":229.313,"e":230.153},{"text":"Routes","s":230.353,"e":230.673},{"text":"to","s":230.713,"e":230.753},{"text":"code,","s":230.833,"e":231.393},{"text":"an","s":231.473,"e":231.553},{"text":"LLM","s":231.633,"e":232.193},{"text":"or","s":232.353,"e":232.433},{"text":"a","s":232.473,"e":232.513},{"text":"person.","s":232.593,"e":233.073},{"text":"A","s":233.473,"e":233.553},{"text":"log","s":233.633,"e":233.953},{"text":"of","s":234.033,"e":234.193},{"text":"every","s":234.273,"e":234.593},{"text":"judgment.","s":234.673,"e":235.233},{"text":"And","s":235.553,"e":235.633},{"text":"an","s":235.713,"e":235.873},{"text":"honest","s":235.953,"e":236.273},{"text":"eval,","s":236.353,"e":237.153},{"text":"held","s":237.273,"e":237.393},{"text":"out","s":237.473,"e":237.633},{"text":"before","s":237.713,"e":238.033},{"text":"the","s":238.073,"e":238.113},{"text":"fix.","s":238.193,"e":238.993}]},{"template":"exercise","chapter":"Build yours","narration":"Now build yours. Point your coding agent at the capstone in the course repository, and have it adapt the pipeline to your own inbox, with your own policy and your own labeled tickets.","params":{"title":"Build your own triage","intro":"Paste this into your coding agent, with the course repo's capstone/ folder available:","prompt":"Using capstone/triage.py as a template, build a triage pipeline for our\nsupport inbox. Put our real policy rules in code where they are exact\n(limits, windows, plans) and in question criteria where they are about\nmeaning. Keep one request per ticket, card and secret redaction in\ncode, and a JSONL log of every answer. Export 300 recent real tickets\nfor labeling, hold out 100 before tuning any question, and report\nroute accuracy with its margin of error, coverage and cost."},"start":239.605,"dur":11.556,"words":[{"text":"Now","s":239.905,"e":240.145},{"text":"build","s":240.225,"e":240.465},{"text":"yours.","s":240.545,"e":241.225},{"text":"Point","s":241.425,"e":241.665},{"text":"your","s":241.745,"e":241.825},{"text":"coding","s":241.985,"e":242.145},{"text":"agent","s":242.225,"e":242.545},{"text":"at","s":242.585,"e":242.625},{"text":"the","s":242.705,"e":242.785},{"text":"capstone","s":242.945,"e":243.345},{"text":"in","s":243.385,"e":243.425},{"text":"the","s":243.465,"e":243.505},{"text":"course","s":243.585,"e":243.745},{"text":"repository,","s":243.825,"e":244.785},{"text":"and","s":244.865,"e":244.945},{"text":"have","s":245.025,"e":245.105},{"text":"it","s":245.185,"e":245.225},{"text":"adapt","s":245.265,"e":245.585},{"text":"the","s":245.665,"e":245.745},{"text":"pipeline","s":245.985,"e":246.225},{"text":"to","s":246.265,"e":246.305},{"text":"your","s":246.385,"e":246.545},{"text":"own","s":246.625,"e":246.785},{"text":"inbox,","s":246.865,"e":247.585},{"text":"with","s":247.665,"e":247.745},{"text":"your","s":247.825,"e":247.905},{"text":"own","s":247.985,"e":248.065},{"text":"policy","s":248.145,"e":248.705},{"text":"and","s":248.785,"e":248.865},{"text":"your","s":248.985,"e":249.105},{"text":"own","s":249.185,"e":249.345},{"text":"labeled","s":249.425,"e":249.745},{"text":"tickets.","s":250.025,"e":250.545}]},{"template":"recap","chapter":"Recap","narration":"So: code prepares, the model judges, code decides. Facts are checked in code before money moves. Bad questions show up in error analysis, and fixes are scored on held-out cases. And twelve for twelve is a smoke test, not proof. That's the capstone.","params":{"items":["Code prepares, the model judges, code decides.","Meaning from the model, facts from code, both before money moves.","Error analysis found 3 bad questions; held-out cases scored the fix.","12/12 is a smoke test: evaluate on hundreds of real tickets."]},"start":251.161,"dur":20.342,"words":[{"text":"So:","s":251.461,"e":252.181},{"text":"code","s":252.261,"e":252.501},{"text":"prepares,","s":252.621,"e":253.461},{"text":"the","s":253.501,"e":253.541},{"text":"model","s":253.621,"e":253.861},{"text":"judges,","s":254.021,"e":254.821},{"text":"code","s":254.901,"e":255.221},{"text":"decides.","s":255.381,"e":256.181},{"text":"Facts","s":256.661,"e":256.981},{"text":"are","s":257.061,"e":257.141},{"text":"checked","s":257.261,"e":257.381},{"text":"in","s":257.461,"e":257.541},{"text":"code","s":257.621,"e":257.941},{"text":"before","s":258.181,"e":258.421},{"text":"money","s":258.501,"e":258.741},{"text":"moves.","s":258.821,"e":259.421},{"text":"Bad","s":259.621,"e":259.781},{"text":"questions","s":259.861,"e":260.341},{"text":"show","s":260.421,"e":260.581},{"text":"up","s":260.661,"e":260.741},{"text":"in","s":260.781,"e":260.821},{"text":"error","s":260.901,"e":261.141},{"text":"analysis,","s":261.221,"e":262.021},{"text":"and","s":262.101,"e":262.181},{"text":"fixes","s":262.261,"e":262.741},{"text":"are","s":262.821,"e":262.901},{"text":"scored","s":262.981,"e":263.301},{"text":"on","s":263.381,"e":263.461},{"text":"held-out","s":263.541,"e":263.941},{"text":"cases.","s":264.021,"e":264.981},{"text":"And","s":265.101,"e":265.221},{"text":"twelve","s":265.381,"e":265.541},{"text":"for","s":265.621,"e":265.781},{"text":"twelve","s":265.861,"e":266.181},{"text":"is","s":266.261,"e":266.341},{"text":"a","s":266.421,"e":266.501},{"text":"smoke","s":266.581,"e":266.901},{"text":"test,","s":266.981,"e":267.541},{"text":"not","s":267.621,"e":267.781},{"text":"proof.","s":267.861,"e":268.661},{"text":"That's","s":268.781,"e":268.941},{"text":"the","s":268.981,"e":269.061},{"text":"capstone.","s":269.181,"e":269.941}]},{"template":"outro","chapter":"The end","narration":"That's the course. Software has always needed decisions, and now it can make the judgments that used to need a person, in a fraction of a second, for a fraction of a cent, with honest probabilities. Use them where they're strong, check them where they're weak, and keep a person close to the decisions that matter. Thanks for watching.","params":{"text":"Now go build software that *decides*.","sub":"Fast judgments, honest probabilities, and code that knows what to do with them.","sub_at":"Use them where they're strong","cast":["snap","mull","curly","you"]},"start":271.503,"dur":20.098,"words":[{"text":"That's","s":271.803,"e":272.123},{"text":"the","s":272.163,"e":272.203},{"text":"course.","s":272.283,"e":273.003},{"text":"Software","s":273.243,"e":273.723},{"text":"has","s":273.883,"e":274.043},{"text":"always","s":274.243,"e":274.443},{"text":"needed","s":274.523,"e":274.763},{"text":"decisions,","s":274.883,"e":275.723},{"text":"and","s":275.803,"e":275.883},{"text":"now","s":275.963,"e":276.203},{"text":"it","s":276.363,"e":276.443},{"text":"can","s":276.523,"e":276.603},{"text":"make","s":276.683,"e":276.763},{"text":"the","s":276.803,"e":276.843},{"text":"judgments","s":276.923,"e":277.323},{"text":"that","s":277.403,"e":277.563},{"text":"used","s":277.643,"e":277.883},{"text":"to","s":277.923,"e":277.963},{"text":"need","s":278.043,"e":278.123},{"text":"a","s":278.163,"e":278.203},{"text":"person,","s":278.283,"e":279.003},{"text":"in","s":279.043,"e":279.083},{"text":"a","s":279.163,"e":279.243},{"text":"fraction","s":279.323,"e":279.643},{"text":"of","s":279.683,"e":279.723},{"text":"a","s":279.763,"e":279.803},{"text":"second,","s":279.883,"e":280.443},{"text":"for","s":280.523,"e":280.603},{"text":"a","s":280.683,"e":280.763},{"text":"fraction","s":280.843,"e":281.163},{"text":"of","s":281.243,"e":281.283},{"text":"a","s":281.323,"e":281.403},{"text":"cent,","s":281.563,"e":282.043},{"text":"with","s":282.163,"e":282.283},{"text":"honest","s":282.323,"e":282.683},{"text":"probabilities.","s":282.763,"e":284.043},{"text":"Use","s":284.123,"e":284.283},{"text":"them","s":284.363,"e":284.443},{"text":"where","s":284.483,"e":284.523},{"text":"they're","s":284.603,"e":284.763},{"text":"strong,","s":285.003,"e":285.483},{"text":"check","s":285.603,"e":285.723},{"text":"them","s":285.763,"e":285.803},{"text":"where","s":285.883,"e":285.963},{"text":"they're","s":286.003,"e":286.123},{"text":"weak,","s":286.283,"e":286.763},{"text":"and","s":286.843,"e":286.923},{"text":"keep","s":287.003,"e":287.083},{"text":"a","s":287.163,"e":287.243},{"text":"person","s":287.323,"e":287.723},{"text":"close","s":287.803,"e":288.043},{"text":"to","s":288.083,"e":288.123},{"text":"the","s":288.203,"e":288.283},{"text":"decisions","s":288.363,"e":288.763},{"text":"that","s":288.843,"e":288.923},{"text":"matter.","s":289.003,"e":289.803},{"text":"Thanks","s":289.883,"e":290.043},{"text":"for","s":290.123,"e":290.203},{"text":"watching.","s":290.283,"e":291.003}]}],"total":292.001,"audio":"/audio/22-capstone.mp3"}