One fast decision in front of expensive things. Here is intent routing (code, a specialist model, or a person), the cascade (cheap model first, escalate only the uncertain), how to tune it so it matches a strong model at a fraction of the cost, and the prompt-cache gotcha that turned one developer's model router into a money loser.
Key takeaways
- Routing: one cheap, fast decision chooses the handler, which may be plain code, a specialist LLM, or a person.
- If you can map the whole decision tree, build it explicitly and optimize each node; don't hand it to an agent loop.
- A cascade answers with the cheap model when it's confident and escalates the rest to a stronger model or a person.
- Tune the cascade on a sample of a few hundred items: pick the cheapest threshold that still agrees with the strong model at your target rate.
- Route per task, not mid-session: switching models mid-conversation throws away the prompt cache.
Try it with your agent
Paste this into your coding agent:
Find our most expensive or slowest LLM call. Put a decision model in front of it: either a router (one Choice for intent + one Score for complexity, sending easy cases to code or a cheap path) or a cascade (accept confident answers, escalate the rest). Tune the threshold on 300 logged inputs using the expensive model's outputs as the oracle, and report agreement, cost and latency at each threshold. Route whole tasks, never switch models mid-session.
Transcript
Intro
Expensive models are great, and wasteful when they're used for everything. The fix is old and simple: put one fast decision in front of the expensive things.
Intent routing
Here's intent routing. A customer message comes in. One Choice decides the intent, and one Score estimates how complex it is, in a single fast call. Order status goes to plain code: a database lookup, no model at all. Product questions go to an LLM with product knowledge. Complaints that are complex, or unclear, go to a person.
Anthropic's Barry Zhang gives the same advice for agents. If you can map out the entire decision tree pretty easily, just build that explicitly, and then optimize every node. It's cheaper, and it gives you more control. A decision model is how you make each node smart.
The cascade
The cascade is routing by confidence. Every item goes to the cheap, fast model first. If it's confident, you keep its answer. If not, the item goes to a stronger, slower model, and only the leftovers reach a person. Most items never touch the expensive part.
The savings can be dramatic. The FrugalGPT paper reported matching GPT-4's performance with up to ninety eight percent lower cost on its benchmarks. And the open model Jeff, on one Apple M4 Max, beat a twenty seven billion parameter model deciding directly, ninety five versus eighty seven percent, in under half a second instead of eight.
Tuning it
How do you set the cascade's threshold? Shreya Shankar's recipe: treat the strong model as the oracle. Run both on a sample, say five hundred items. Simulate each threshold, measuring how often the cascade agrees with the oracle, and what it costs. Throw out every threshold below your target, and pick the cheapest one left. And if confidence carries no signal, the only threshold left sends everything to the oracle. The procedure tells you there's nothing to save, instead of pretending.
Routing effort
You can route effort, not just models. The Jeff team put a small decision model inside a coding agent, deciding at each step whether the big model should think hard. Over twelve hundred tasks, it kept the same pass rate, sixty two percent, in about two thirds of the time. Simply turning thinking off was faster too, but seven and a half points worse.
The cache gotcha
One gotcha from Hacker News. A developer used a decision model to route each turn of a session between small and large models, and lost money. Swapping models mid-session invalidates the prompt cache, which is where a lot of the savings live. Route whole tasks, or route the effort level, instead.
Quick check
Quick check. In your cascade, the decision model's answers above point nine agree with the strong model only as often as its answers below point five. What does that tell you?
Try it
Try it. Have your agent put a router or a cascade in front of your most expensive model call.
Recap
So: one fast decision in front of expensive things. Route to code, a specialist, or a person. Cascade: cheap first, escalate the uncertain, tuned against a strong model. Route effort, not just models, and never mid-session. Next: getting exact values out, without generating them.