{"id":"14-route","number":14,"slug":"routing-and-cascades","title":"Route and cascade","short":"Route and cascade","module":"Composing judgments","summary":"One fast decision in front of expensive things. Here is intent routing (code, a specialist model, or a person), the cascade (cheap model first, escalate only the uncertain), how to tune it so it matches a strong model at a fraction of the cost, and the prompt-cache gotcha that turned one developer's model router into a money loser.","voice":"Jessica","scenes":[{"template":"title","chapter":"Intro","narration":"Expensive models are great, and wasteful when they're used for everything. The fix is old and simple: put one fast decision in front of the expensive things.","params":{"kicker":"Lesson 14","module":"Module 4 · Composing judgments","title":"Route and *cascade*","subtitle":"One fast decision in front of expensive things.","cast":["snap",{"who":"mull","mood":"tired"}]},"start":0.5,"dur":12.433,"words":[{"text":"Expensive","s":0.8,"e":1.36},{"text":"models","s":1.44,"e":1.76},{"text":"are","s":1.84,"e":2},{"text":"great,","s":2.24,"e":2.96},{"text":"and","s":3.16,"e":3.36},{"text":"wasteful","s":3.6,"e":4.24},{"text":"when","s":4.48,"e":4.64},{"text":"they're","s":4.693,"e":4.8},{"text":"used","s":4.92,"e":5.04},{"text":"for","s":5.16,"e":5.28},{"text":"everything.","s":5.36,"e":6.48},{"text":"The","s":6.52,"e":6.56},{"text":"fix","s":6.64,"e":6.96},{"text":"is","s":7.04,"e":7.2},{"text":"old","s":7.28,"e":7.6},{"text":"and","s":7.64,"e":7.68},{"text":"simple:","s":7.84,"e":8.64},{"text":"put","s":8.8,"e":8.96},{"text":"one","s":9.04,"e":9.2},{"text":"fast","s":9.36,"e":9.6},{"text":"decision","s":9.88,"e":10.16},{"text":"in","s":10.24,"e":10.32},{"text":"front","s":10.4,"e":10.48},{"text":"of","s":10.56,"e":10.64},{"text":"the","s":10.68,"e":10.72},{"text":"expensive","s":10.84,"e":11.28},{"text":"things.","s":11.52,"e":12.08}]},{"template":"router","chapter":"Intent routing","narration":"Here's intent routing. A customer message comes in. One Choice decides the intent, and one Score estimates how complex it is, in a single fast call. Order status goes to plain code: a database lookup, no model at all. Product questions go to an LLM with product knowledge. Complaints that are complex, or unclear, go to a person.","params":{"source":{"label":"customer message","text":"Where is my order A-104? It was due Monday."},"hub":{"label":"intent + complexity","sub":"one request, ~0.3 s"},"routes":[{"label":"Order lookup","sub":"plain code, no model","icon":"db","tone":"ink","at":"plain code"},{"label":"Product specialist","sub":"an LLM with product docs","who":"mull","tone":"s2","at":"Product questions"},{"label":"A person","sub":"complex or unclear","who":"you","tone":"human","at":"go to a person"}]},"start":12.933,"dur":22.579,"words":[{"text":"Here's","s":13.233,"e":13.553},{"text":"intent","s":13.633,"e":14.113},{"text":"routing.","s":14.313,"e":14.593},{"text":"A","s":14.993,"e":15.073},{"text":"customer","s":15.273,"e":15.473},{"text":"message","s":15.553,"e":15.873},{"text":"comes","s":15.953,"e":16.193},{"text":"in.","s":16.273,"e":16.753},{"text":"One","s":16.993,"e":17.153},{"text":"Choice","s":17.233,"e":17.473},{"text":"decides","s":17.553,"e":17.953},{"text":"the","s":18.033,"e":18.073},{"text":"intent,","s":18.113,"e":18.753},{"text":"and","s":18.833,"e":18.913},{"text":"one","s":18.993,"e":19.153},{"text":"Score","s":19.233,"e":19.633},{"text":"estimates","s":19.713,"e":20.193},{"text":"how","s":20.273,"e":20.353},{"text":"complex","s":20.433,"e":20.833},{"text":"it","s":20.873,"e":20.913},{"text":"is,","s":20.993,"e":21.393},{"text":"in","s":21.473,"e":21.553},{"text":"a","s":21.593,"e":21.633},{"text":"single","s":21.793,"e":22.113},{"text":"fast","s":22.353,"e":22.673},{"text":"call.","s":22.753,"e":23.153},{"text":"Order","s":23.553,"e":23.793},{"text":"status","s":23.873,"e":24.273},{"text":"goes","s":24.313,"e":24.513},{"text":"to","s":24.553,"e":24.593},{"text":"plain","s":24.673,"e":24.913},{"text":"code:","s":24.993,"e":25.553},{"text":"a","s":25.593,"e":25.633},{"text":"database","s":25.873,"e":26.113},{"text":"lookup,","s":26.313,"e":26.753},{"text":"no","s":26.833,"e":26.913},{"text":"model","s":26.993,"e":27.233},{"text":"at","s":27.313,"e":27.393},{"text":"all.","s":27.513,"e":27.713},{"text":"Product","s":28.033,"e":28.353},{"text":"questions","s":28.433,"e":28.913},{"text":"go","s":28.993,"e":29.073},{"text":"to","s":29.153,"e":29.233},{"text":"an","s":29.313,"e":29.393},{"text":"LLM","s":29.473,"e":29.953},{"text":"with","s":30.033,"e":30.113},{"text":"product","s":30.193,"e":30.433},{"text":"knowledge.","s":30.513,"e":31.153},{"text":"Complaints","s":31.313,"e":31.753},{"text":"that","s":31.793,"e":31.873},{"text":"are","s":31.913,"e":31.953},{"text":"complex,","s":32.033,"e":32.673},{"text":"or","s":32.753,"e":32.833},{"text":"unclear,","s":32.913,"e":33.713},{"text":"go","s":33.793,"e":33.873},{"text":"to","s":33.953,"e":34.033},{"text":"a","s":34.073,"e":34.113},{"text":"person.","s":34.193,"e":34.913}]},{"template":"quote","chapter":"Intent routing","narration":"Anthropic's Barry Zhang gives the same advice for agents. If you can map out the entire decision tree pretty easily, just build that explicitly, and then optimize every node. It's cheaper, and it gives you more control. A decision model is how you make each node smart.","params":{"text":"If you can map out the entire decision tree pretty easily, *just build that explicitly* and then optimize every node.","who":"Barry Zhang","role":"Anthropic","src":"How We Build Effective Agents","t":171},"start":35.512,"dur":18.112,"words":[{"text":"Anthropic's","s":35.812,"e":36.452},{"text":"Barry","s":36.532,"e":36.772},{"text":"Zhang","s":36.852,"e":37.172},{"text":"gives","s":37.252,"e":37.412},{"text":"the","s":37.492,"e":37.572},{"text":"same","s":37.652,"e":37.812},{"text":"advice","s":37.892,"e":38.212},{"text":"for","s":38.292,"e":38.372},{"text":"agents.","s":38.452,"e":39.212},{"text":"If","s":39.412,"e":39.492},{"text":"you","s":39.572,"e":39.652},{"text":"can","s":39.732,"e":39.812},{"text":"map","s":39.932,"e":40.052},{"text":"out","s":40.132,"e":40.292},{"text":"the","s":40.332,"e":40.372},{"text":"entire","s":40.532,"e":40.932},{"text":"decision","s":41.012,"e":41.332},{"text":"tree","s":41.412,"e":41.652},{"text":"pretty","s":41.732,"e":41.972},{"text":"easily,","s":42.052,"e":42.852},{"text":"just","s":42.972,"e":43.092},{"text":"build","s":43.252,"e":43.492},{"text":"that","s":43.572,"e":43.652},{"text":"explicitly,","s":43.932,"e":44.692},{"text":"and","s":44.772,"e":44.852},{"text":"then","s":44.972,"e":45.092},{"text":"optimize","s":45.172,"e":45.732},{"text":"every","s":45.892,"e":46.132},{"text":"node.","s":46.212,"e":46.932},{"text":"It's","s":47.012,"e":47.172},{"text":"cheaper,","s":47.252,"e":47.812},{"text":"and","s":47.932,"e":48.052},{"text":"it","s":48.092,"e":48.132},{"text":"gives","s":48.212,"e":48.372},{"text":"you","s":48.412,"e":48.452},{"text":"more","s":48.532,"e":48.612},{"text":"control.","s":48.692,"e":49.452},{"text":"A","s":49.732,"e":49.812},{"text":"decision","s":50.012,"e":50.212},{"text":"model","s":50.292,"e":50.612},{"text":"is","s":50.692,"e":50.772},{"text":"how","s":50.852,"e":51.012},{"text":"you","s":51.092,"e":51.172},{"text":"make","s":51.332,"e":51.492},{"text":"each","s":51.572,"e":51.732},{"text":"node","s":51.892,"e":52.132},{"text":"smart.","s":52.212,"e":53.012}]},{"template":"flow","chapter":"The cascade","narration":"The cascade is routing by confidence. Every item goes to the cheap, fast model first. If it's confident, you keep its answer. If not, the item goes to a stronger, slower model, and only the leftovers reach a person. Most items never touch the expensive part.","params":{"title":"Cheap first, escalate the uncertain","nodes":[{"label":"Decision model","sub":"every item · cents","kind":"snap","edge":"unsure","at":"cheap, fast model"},{"label":"Reasoning model","sub":"the uncertain few","kind":"mull","edge":"still unsure","at":"stronger, slower"},{"label":"A person","sub":"the leftovers","kind":"you","at":"leftovers"}],"caption":"Confident answers exit at each step.","caption_at":"never touch"},"start":53.624,"dur":18.348,"words":[{"text":"The","s":53.924,"e":54.084},{"text":"cascade","s":54.364,"e":54.644},{"text":"is","s":54.724,"e":54.804},{"text":"routing","s":54.884,"e":55.204},{"text":"by","s":55.284,"e":55.444},{"text":"confidence.","s":55.524,"e":56.644},{"text":"Every","s":56.804,"e":56.964},{"text":"item","s":57.044,"e":57.284},{"text":"goes","s":57.364,"e":57.524},{"text":"to","s":57.604,"e":57.684},{"text":"the","s":57.764,"e":57.844},{"text":"cheap,","s":57.924,"e":58.404},{"text":"fast","s":58.564,"e":58.724},{"text":"model","s":58.804,"e":59.124},{"text":"first.","s":59.204,"e":60.004},{"text":"If","s":60.044,"e":60.084},{"text":"it's","s":60.164,"e":60.244},{"text":"confident,","s":60.324,"e":61.044},{"text":"you","s":61.124,"e":61.204},{"text":"keep","s":61.284,"e":61.364},{"text":"its","s":61.444,"e":61.524},{"text":"answer.","s":61.724,"e":62.324},{"text":"If","s":62.404,"e":62.484},{"text":"not,","s":62.564,"e":63.204},{"text":"the","s":63.244,"e":63.284},{"text":"item","s":63.364,"e":63.604},{"text":"goes","s":63.684,"e":63.924},{"text":"to","s":63.964,"e":64.004},{"text":"a","s":64.084,"e":64.164},{"text":"stronger,","s":64.244,"e":64.964},{"text":"slower","s":65.044,"e":65.364},{"text":"model,","s":65.444,"e":65.924},{"text":"and","s":66.044,"e":66.164},{"text":"only","s":66.244,"e":66.484},{"text":"the","s":66.564,"e":66.644},{"text":"leftovers","s":66.724,"e":67.284},{"text":"reach","s":67.404,"e":67.524},{"text":"a","s":67.564,"e":67.604},{"text":"person.","s":67.844,"e":68.564},{"text":"Most","s":68.684,"e":68.804},{"text":"items","s":68.884,"e":69.284},{"text":"never","s":69.524,"e":69.764},{"text":"touch","s":69.844,"e":70.084},{"text":"the","s":70.124,"e":70.164},{"text":"expensive","s":70.244,"e":70.724},{"text":"part.","s":70.804,"e":71.364}]},{"template":"stat","chapter":"The cascade","narration":"The savings can be dramatic. The FrugalGPT paper reported matching GPT-4's performance with up to ninety eight percent lower cost on its benchmarks. And the open model Jeff, on one Apple M4 Max, beat a twenty seven billion parameter model deciding directly, ninety five versus eighty seven percent, in under half a second instead of eight.","params":{"items":[{"value":98,"suffix":"%","label":"max cost cut at GPT-4 quality (FrugalGPT, 2023)","tone":"agent","at":"ninety eight percent"},{"value":95,"decimals":1,"suffix":"%","label":"Jeff cascade, 0.46 s per decision","tone":"good","at":"ninety five"},{"value":86.6,"decimals":1,"suffix":"%","label":"27B model alone, 8.1 s","tone":"bad","at":"eighty seven"}],"src":"Authors' own measurements (FrugalGPT paper; Jeff README, Apple M4 Max).","punch":false},"start":71.972,"dur":24.121,"words":[{"text":"The","s":72.272,"e":72.432},{"text":"savings","s":72.512,"e":72.992},{"text":"can","s":73.072,"e":73.152},{"text":"be","s":73.232,"e":73.312},{"text":"dramatic.","s":73.392,"e":74.272},{"text":"The","s":74.672,"e":74.752},{"text":"FrugalGPT","s":74.832,"e":75.712},{"text":"paper","s":75.792,"e":76.112},{"text":"reported","s":76.192,"e":76.672},{"text":"matching","s":76.832,"e":76.992},{"text":"GPT-4's","s":77.152,"e":77.872},{"text":"performance","s":77.952,"e":78.512},{"text":"with","s":78.592,"e":78.752},{"text":"up","s":78.832,"e":78.912},{"text":"to","s":78.992,"e":79.152},{"text":"ninety","s":79.232,"e":79.552},{"text":"eight","s":79.712,"e":79.792},{"text":"percent","s":79.952,"e":80.272},{"text":"lower","s":80.432,"e":80.672},{"text":"cost","s":80.752,"e":81.152},{"text":"on","s":81.192,"e":81.232},{"text":"its","s":81.312,"e":81.392},{"text":"benchmarks.","s":81.472,"e":82.352},{"text":"And","s":82.592,"e":82.672},{"text":"the","s":82.752,"e":82.832},{"text":"open","s":82.912,"e":83.152},{"text":"model","s":83.272,"e":83.392},{"text":"Jeff,","s":83.552,"e":84.112},{"text":"on","s":84.232,"e":84.352},{"text":"one","s":84.432,"e":84.592},{"text":"Apple","s":84.752,"e":84.992},{"text":"M4","s":85.072,"e":85.392},{"text":"Max,","s":85.472,"e":86.112},{"text":"beat","s":86.192,"e":86.352},{"text":"a","s":86.432,"e":86.512},{"text":"twenty","s":86.672,"e":86.832},{"text":"seven","s":86.992,"e":87.152},{"text":"billion","s":87.392,"e":87.792},{"text":"parameter","s":87.872,"e":88.272},{"text":"model","s":88.352,"e":88.592},{"text":"deciding","s":88.672,"e":89.072},{"text":"directly,","s":89.152,"e":90.112},{"text":"ninety","s":90.192,"e":90.432},{"text":"five","s":90.512,"e":90.752},{"text":"versus","s":90.832,"e":91.312},{"text":"eighty","s":91.392,"e":91.552},{"text":"seven","s":91.632,"e":91.792},{"text":"percent,","s":91.952,"e":92.672},{"text":"in","s":92.752,"e":92.832},{"text":"under","s":92.912,"e":93.152},{"text":"half","s":93.232,"e":93.392},{"text":"a","s":93.472,"e":93.552},{"text":"second","s":93.632,"e":94.032},{"text":"instead","s":94.272,"e":94.592},{"text":"of","s":94.712,"e":94.832},{"text":"eight.","s":94.992,"e":95.472}]},{"template":"list","chapter":"Tuning it","narration":"How do you set the cascade's threshold? Shreya Shankar's recipe: treat the strong model as the oracle. Run both on a sample, say five hundred items. Simulate each threshold, measuring how often the cascade agrees with the oracle, and what it costs. Throw out every threshold below your target, and pick the cheapest one left. And if confidence carries no signal, the only threshold left sends everything to the oracle. The procedure tells you there's nothing to save, instead of pretending.","params":{"title":"Tune against the strong model","style":"num","items":[{"text":"The strong model's answers are the **oracle**","at":"oracle"},{"text":"Run both on a sample of **~500** items","at":"five hundred"},{"text":"For each threshold: **agreement** with the oracle and **cost**","at":"Simulate each threshold"},{"text":"Drop thresholds below target; pick the **cheapest** left","at":"cheapest"},{"text":"Only “send everything” qualifies? Confidence has **no signal**","at":"only threshold left"}]},"start":96.093,"dur":31.226,"words":[{"text":"How","s":96.393,"e":96.633},{"text":"do","s":96.713,"e":96.753},{"text":"you","s":96.793,"e":96.953},{"text":"set","s":97.033,"e":97.113},{"text":"the","s":97.193,"e":97.233},{"text":"cascade's","s":97.273,"e":97.913},{"text":"threshold?","s":98.233,"e":98.633},{"text":"Shreya","s":99.193,"e":99.513},{"text":"Shankar's","s":99.593,"e":100.073},{"text":"recipe:","s":100.153,"e":101.113},{"text":"treat","s":101.153,"e":101.353},{"text":"the","s":101.393,"e":101.433},{"text":"strong","s":101.593,"e":101.913},{"text":"model","s":102.113,"e":102.313},{"text":"as","s":102.393,"e":102.473},{"text":"the","s":102.513,"e":102.553},{"text":"oracle.","s":102.713,"e":103.593},{"text":"Run","s":103.713,"e":103.833},{"text":"both","s":103.913,"e":104.153},{"text":"on","s":104.193,"e":104.233},{"text":"a","s":104.313,"e":104.393},{"text":"sample,","s":104.593,"e":105.033},{"text":"say","s":105.193,"e":105.353},{"text":"five","s":105.513,"e":105.673},{"text":"hundred","s":105.753,"e":105.913},{"text":"items.","s":105.993,"e":106.793},{"text":"Simulate","s":106.873,"e":107.353},{"text":"each","s":107.433,"e":107.593},{"text":"threshold,","s":107.673,"e":108.393},{"text":"measuring","s":108.433,"e":108.793},{"text":"how","s":108.873,"e":109.113},{"text":"often","s":109.273,"e":109.433},{"text":"the","s":109.473,"e":109.513},{"text":"cascade","s":109.593,"e":109.993},{"text":"agrees","s":110.073,"e":110.473},{"text":"with","s":110.553,"e":110.633},{"text":"the","s":110.673,"e":110.713},{"text":"oracle,","s":110.793,"e":111.433},{"text":"and","s":111.553,"e":111.673},{"text":"what","s":111.753,"e":111.833},{"text":"it","s":111.873,"e":111.913},{"text":"costs.","s":111.993,"e":112.873},{"text":"Throw","s":112.993,"e":113.113},{"text":"out","s":113.193,"e":113.433},{"text":"every","s":113.513,"e":113.753},{"text":"threshold","s":113.833,"e":114.233},{"text":"below","s":114.313,"e":114.633},{"text":"your","s":114.713,"e":114.793},{"text":"target,","s":114.873,"e":115.353},{"text":"and","s":115.433,"e":115.513},{"text":"pick","s":115.593,"e":115.673},{"text":"the","s":115.713,"e":115.753},{"text":"cheapest","s":115.913,"e":116.233},{"text":"one","s":116.313,"e":116.393},{"text":"left.","s":116.473,"e":117.193},{"text":"And","s":117.233,"e":117.273},{"text":"if","s":117.353,"e":117.433},{"text":"confidence","s":117.513,"e":117.913},{"text":"carries","s":117.993,"e":118.313},{"text":"no","s":118.393,"e":118.553},{"text":"signal,","s":118.713,"e":119.353},{"text":"the","s":119.473,"e":119.593},{"text":"only","s":119.713,"e":119.833},{"text":"threshold","s":119.913,"e":120.313},{"text":"left","s":120.393,"e":120.713},{"text":"sends","s":120.793,"e":121.193},{"text":"everything","s":121.273,"e":121.673},{"text":"to","s":121.713,"e":121.753},{"text":"the","s":121.833,"e":121.913},{"text":"oracle.","s":122.153,"e":122.793},{"text":"The","s":122.833,"e":122.873},{"text":"procedure","s":123.113,"e":123.353},{"text":"tells","s":123.393,"e":123.593},{"text":"you","s":123.673,"e":123.753},{"text":"there's","s":123.833,"e":124.073},{"text":"nothing","s":124.153,"e":124.393},{"text":"to","s":124.473,"e":124.553},{"text":"save,","s":124.633,"e":125.273},{"text":"instead","s":125.473,"e":125.673},{"text":"of","s":125.713,"e":125.753},{"text":"pretending.","s":125.833,"e":126.713}]},{"template":"statement","chapter":"Routing effort","narration":"You can route effort, not just models. The Jeff team put a small decision model inside a coding agent, deciding at each step whether the big model should think hard. Over twelve hundred tasks, it kept the same pass rate, sixty two percent, in about two thirds of the time. Simply turning thinking off was faster too, but seven and a half points worse.","params":{"eyebrow":"Jeff-Code: a decision model decides when to think","text":"Same pass rate (62.4% vs 62.8%), *32% less time*.","sub":"Turning thinking off entirely: 7.6 points worse. The fast judge is what keeps quality.","sub_at":"turning thinking off","art":"clock","tone":"agent","punch":"two thirds of the time"},"start":127.319,"dur":24.016,"words":[{"text":"You","s":127.619,"e":127.779},{"text":"can","s":127.859,"e":127.939},{"text":"route","s":128.099,"e":128.259},{"text":"effort,","s":128.339,"e":129.059},{"text":"not","s":129.139,"e":129.299},{"text":"just","s":129.379,"e":129.539},{"text":"models.","s":129.819,"e":130.579},{"text":"The","s":130.619,"e":130.659},{"text":"Jeff","s":130.739,"e":130.979},{"text":"team","s":131.059,"e":131.299},{"text":"put","s":131.379,"e":131.459},{"text":"a","s":131.539,"e":131.619},{"text":"small","s":131.739,"e":131.859},{"text":"decision","s":131.939,"e":132.339},{"text":"model","s":132.419,"e":132.659},{"text":"inside","s":132.739,"e":133.219},{"text":"a","s":133.299,"e":133.379},{"text":"coding","s":133.539,"e":133.699},{"text":"agent,","s":133.779,"e":134.419},{"text":"deciding","s":134.499,"e":134.899},{"text":"at","s":134.979,"e":135.059},{"text":"each","s":135.139,"e":135.299},{"text":"step","s":135.379,"e":135.699},{"text":"whether","s":135.779,"e":136.019},{"text":"the","s":136.059,"e":136.099},{"text":"big","s":136.259,"e":136.419},{"text":"model","s":136.499,"e":136.739},{"text":"should","s":136.819,"e":137.059},{"text":"think","s":137.179,"e":137.299},{"text":"hard.","s":137.379,"e":138.259},{"text":"Over","s":138.419,"e":138.579},{"text":"twelve","s":138.659,"e":138.899},{"text":"hundred","s":139.059,"e":139.299},{"text":"tasks,","s":139.459,"e":140.179},{"text":"it","s":140.259,"e":140.339},{"text":"kept","s":140.419,"e":140.499},{"text":"the","s":140.579,"e":140.739},{"text":"same","s":140.899,"e":141.059},{"text":"pass","s":141.219,"e":141.379},{"text":"rate,","s":141.459,"e":141.859},{"text":"sixty","s":141.939,"e":142.179},{"text":"two","s":142.259,"e":142.299},{"text":"percent,","s":142.339,"e":143.059},{"text":"in","s":143.139,"e":143.219},{"text":"about","s":143.379,"e":143.539},{"text":"two","s":143.619,"e":143.699},{"text":"thirds","s":143.859,"e":144.099},{"text":"of","s":144.139,"e":144.179},{"text":"the","s":144.259,"e":144.339},{"text":"time.","s":144.539,"e":145.219},{"text":"Simply","s":145.419,"e":145.619},{"text":"turning","s":145.739,"e":146.019},{"text":"thinking","s":146.179,"e":146.579},{"text":"off","s":146.659,"e":146.979},{"text":"was","s":147.099,"e":147.219},{"text":"faster","s":147.259,"e":147.619},{"text":"too,","s":147.699,"e":148.339},{"text":"but","s":148.419,"e":148.739},{"text":"seven","s":148.899,"e":149.059},{"text":"and","s":149.139,"e":149.219},{"text":"a","s":149.259,"e":149.299},{"text":"half","s":149.379,"e":149.459},{"text":"points","s":149.539,"e":149.859},{"text":"worse.","s":149.939,"e":150.739}]},{"template":"statement","chapter":"The cache gotcha","narration":"One gotcha from Hacker News. A developer used a decision model to route each turn of a session between small and large models, and lost money. Swapping models mid-session invalidates the prompt cache, which is where a lot of the savings live. Route whole tasks, or route the effort level, instead.","params":{"eyebrow":"Hacker News, “Jev in 25 lines of Python” thread","text":"Mid-session swapping *invalidates the cache*.","sub":"Route whole tasks, or the reasoning effort, not every turn.","sub_at":"Route whole tasks","art":"warn","tone":"bad","punch":"lost money"},"start":151.335,"dur":20.098,"words":[{"text":"One","s":151.635,"e":151.875},{"text":"gotcha","s":151.955,"e":152.435},{"text":"from","s":152.515,"e":152.595},{"text":"Hacker","s":152.675,"e":152.915},{"text":"News.","s":152.995,"e":153.715},{"text":"A","s":153.955,"e":154.035},{"text":"developer","s":154.275,"e":154.515},{"text":"used","s":154.595,"e":154.835},{"text":"a","s":154.915,"e":154.955},{"text":"decision","s":154.995,"e":155.395},{"text":"model","s":155.475,"e":155.795},{"text":"to","s":155.875,"e":155.955},{"text":"route","s":156.035,"e":156.275},{"text":"each","s":156.355,"e":156.595},{"text":"turn","s":156.675,"e":156.835},{"text":"of","s":156.915,"e":156.995},{"text":"a","s":157.048,"e":157.102},{"text":"session","s":157.155,"e":157.555},{"text":"between","s":157.715,"e":157.875},{"text":"small","s":158.035,"e":158.435},{"text":"and","s":158.515,"e":158.595},{"text":"large","s":158.675,"e":158.915},{"text":"models,","s":158.995,"e":159.635},{"text":"and","s":159.715,"e":159.795},{"text":"lost","s":159.955,"e":160.115},{"text":"money.","s":160.195,"e":161.075},{"text":"Swapping","s":161.155,"e":161.475},{"text":"models","s":161.555,"e":161.875},{"text":"mid-session","s":161.955,"e":162.675},{"text":"invalidates","s":162.955,"e":163.395},{"text":"the","s":163.475,"e":163.555},{"text":"prompt","s":163.635,"e":163.875},{"text":"cache,","s":163.955,"e":164.515},{"text":"which","s":164.635,"e":164.755},{"text":"is","s":164.835,"e":164.915},{"text":"where","s":164.995,"e":165.075},{"text":"a","s":165.155,"e":165.235},{"text":"lot","s":165.315,"e":165.475},{"text":"of","s":165.515,"e":165.555},{"text":"the","s":165.635,"e":165.715},{"text":"savings","s":165.755,"e":166.115},{"text":"live.","s":166.195,"e":166.995},{"text":"Route","s":167.155,"e":167.315},{"text":"whole","s":167.395,"e":167.555},{"text":"tasks,","s":167.635,"e":168.355},{"text":"or","s":168.475,"e":168.595},{"text":"route","s":168.835,"e":168.995},{"text":"the","s":169.075,"e":169.155},{"text":"effort","s":169.235,"e":169.475},{"text":"level,","s":169.555,"e":169.955},{"text":"instead.","s":170.235,"e":170.835}]},{"template":"quiz","chapter":"Quick check","narration":"Quick check. In your cascade, the decision model's answers above point nine agree with the strong model only as often as its answers below point five. What does that tell you?","params":{"q":"Answers above 0.9 agree with the oracle **no more often** than answers below 0.5. What now?","options":["Raise the threshold to 0.99","Confidence carries no signal on this task: don't cascade it this way","Lower the threshold to 0.5","Add more questions to the request"],"answer":1,"explain":"A cascade only works if confidence tracks correctness. Here it doesn't, so any threshold routes at random. Improve the questions or the model, or send everything to the strong model."},"start":171.433,"dur":16.576,"words":[{"text":"Quick","s":171.733,"e":171.973},{"text":"check.","s":172.053,"e":172.573},{"text":"In","s":172.773,"e":172.853},{"text":"your","s":172.933,"e":173.013},{"text":"cascade,","s":173.093,"e":173.893},{"text":"the","s":173.973,"e":174.013},{"text":"decision","s":174.053,"e":174.453},{"text":"model's","s":174.533,"e":174.853},{"text":"answers","s":174.933,"e":175.333},{"text":"above","s":175.413,"e":175.733},{"text":"point","s":175.813,"e":176.053},{"text":"nine","s":176.133,"e":176.453},{"text":"agree","s":176.613,"e":176.933},{"text":"with","s":177.013,"e":177.093},{"text":"the","s":177.133,"e":177.173},{"text":"strong","s":177.253,"e":177.573},{"text":"model","s":177.733,"e":177.893},{"text":"only","s":178.133,"e":178.453},{"text":"as","s":178.533,"e":178.613},{"text":"often","s":178.693,"e":179.013},{"text":"as","s":179.093,"e":179.173},{"text":"its","s":179.253,"e":179.333},{"text":"answers","s":179.413,"e":179.813},{"text":"below","s":179.973,"e":180.133},{"text":"point","s":180.213,"e":180.533},{"text":"five.","s":180.613,"e":181.573},{"text":"What","s":181.653,"e":181.733},{"text":"does","s":181.813,"e":181.893},{"text":"that","s":181.973,"e":182.053},{"text":"tell","s":182.133,"e":182.293},{"text":"you?","s":182.373,"e":182.853}]},{"template":"exercise","chapter":"Try it","narration":"Try it. Have your agent put a router or a cascade in front of your most expensive model call.","params":{"title":"Put a judge in front","intro":"Paste this into your coding agent:","prompt":"Find our most expensive or slowest LLM call. Put a decision model in\nfront of it: either a router (one Choice for intent + one Score for\ncomplexity, sending easy cases to code or a cheap path) or a cascade\n(accept confident answers, escalate the rest). Tune the threshold on\n300 logged inputs using the expensive model's outputs as the oracle,\nand report agreement, cost and latency at each threshold. Route whole\ntasks, never switch models mid-session."},"start":188.009,"dur":6.984,"words":[{"text":"Try","s":188.309,"e":188.629},{"text":"it.","s":188.709,"e":189.069},{"text":"Have","s":189.269,"e":189.429},{"text":"your","s":189.509,"e":189.589},{"text":"agent","s":189.669,"e":189.909},{"text":"put","s":189.989,"e":190.069},{"text":"a","s":190.149,"e":190.229},{"text":"router","s":190.469,"e":190.709},{"text":"or","s":190.789,"e":190.949},{"text":"a","s":190.989,"e":191.029},{"text":"cascade","s":191.109,"e":191.669},{"text":"in","s":191.749,"e":191.829},{"text":"front","s":191.909,"e":192.069},{"text":"of","s":192.109,"e":192.149},{"text":"your","s":192.269,"e":192.389},{"text":"most","s":192.549,"e":192.709},{"text":"expensive","s":192.789,"e":193.429},{"text":"model","s":193.509,"e":193.749},{"text":"call.","s":193.829,"e":194.389}]},{"template":"recap","chapter":"Recap","narration":"So: one fast decision in front of expensive things. Route to code, a specialist, or a person. Cascade: cheap first, escalate the uncertain, tuned against a strong model. Route effort, not just models, and never mid-session. Next: getting exact values out, without generating them.","params":{"items":["One fast decision picks the handler: code, LLM, or person.","Cascade: accept confident answers, escalate the rest.","Tune thresholds against the strong model on a ~500-item sample.","Route whole tasks or effort; mid-session swaps kill the cache."],"next":"Select, don't generate","next_at":"Next"},"start":194.993,"dur":23.686,"words":[{"text":"So:","s":195.293,"e":196.093},{"text":"one","s":196.173,"e":196.333},{"text":"fast","s":196.413,"e":196.653},{"text":"decision","s":196.733,"e":197.213},{"text":"in","s":197.293,"e":197.373},{"text":"front","s":197.453,"e":197.613},{"text":"of","s":197.653,"e":197.693},{"text":"expensive","s":197.773,"e":198.413},{"text":"things.","s":198.493,"e":199.213},{"text":"Route","s":199.453,"e":199.693},{"text":"to","s":199.733,"e":199.773},{"text":"code,","s":199.853,"e":200.493},{"text":"a","s":200.533,"e":200.573},{"text":"specialist,","s":200.653,"e":201.533},{"text":"or","s":201.573,"e":201.613},{"text":"a","s":201.693,"e":201.733},{"text":"person.","s":201.773,"e":202.333},{"text":"Cascade:","s":202.893,"e":203.933},{"text":"cheap","s":204.013,"e":204.253},{"text":"first,","s":204.333,"e":205.053},{"text":"escalate","s":205.253,"e":205.533},{"text":"the","s":205.613,"e":205.653},{"text":"uncertain,","s":205.693,"e":206.653},{"text":"tuned","s":206.693,"e":206.973},{"text":"against","s":207.053,"e":207.373},{"text":"a","s":207.413,"e":207.453},{"text":"strong","s":207.613,"e":207.933},{"text":"model.","s":208.013,"e":208.413},{"text":"Route","s":208.893,"e":209.053},{"text":"effort,","s":209.213,"e":209.853},{"text":"not","s":209.933,"e":210.093},{"text":"just","s":210.173,"e":210.333},{"text":"models,","s":210.613,"e":210.973},{"text":"and","s":211.053,"e":211.133},{"text":"never","s":211.293,"e":211.613},{"text":"mid-session.","s":211.693,"e":212.413},{"text":"Next:","s":213.453,"e":214.013},{"text":"getting","s":214.173,"e":214.253},{"text":"exact","s":214.653,"e":214.733},{"text":"values","s":215.133,"e":215.213},{"text":"out,","s":215.453,"e":215.813},{"text":"without","s":216.013,"e":216.173},{"text":"generating","s":216.333,"e":216.653},{"text":"them.","s":216.893,"e":217.133}]}],"total":219.079,"audio":"/audio/14-route.mp3"}