The whole course in one pipeline you can run: code prepares the state, one request asks fifteen questions, and code routes every ticket to a refund, a reset link, an LLM, a log, a person or a block. Here is how we built it, the three bad questions our first run exposed, what held-out tickets showed, and why twelve out of twelve still proves less than it sounds.
Key takeaways
- The pipeline: code prepares (redacts cards, lists candidate orders), one request asks 15 questions, code decides and logs.
- The model judges meaning; code checks facts: a refund needs the model to hear a duplicate charge and the records to show one, under the $100 limit.
- Our first run routed 14 of 24 tickets right; three badly written questions caused every miss.
- After fixing them, 24 of 24, and 12 of 12 on tickets written before the fix; 16 of 36 needed no person.
- Twelve out of twelve still allows a true error rate up to about 25%: evaluate on hundreds of real tickets before trusting it.
Try it with your agent
Paste this into your coding agent, with the course repo's capstone/ folder available:
Using capstone/triage.py as a template, build a triage pipeline for our support inbox. Put our real policy rules in code where they are exact (limits, windows, plans) and in question criteria where they are about meaning. Keep one request per ticket, card and secret redaction in code, and a JSONL log of every answer. Export 300 recent real tickets for labeling, hold out 100 before tuning any question, and report route accuracy with its margin of error, coverage and cost.
Transcript
Intro
Time to put it all together. One pipeline, built from everything in this course, that triages a support inbox. The code is in the course repository, and you can run it yourself.
The design
Here's the design. First, code prepares the state. It redacts card numbers, so no model ever sees one, and lists the customer's orders as candidates. Then one request asks fifteen questions at once, in a quarter of a second. Then code decides: thresholds, policy checks, and a log of every answer.
Here's one ticket. My card, four one one one and so on, was charged twice for order P-4410. The card number is already gone. The answers: billing, at one point oh. A duplicate charge, point nine nine. The order, picked from the customer's own orders: P-4410. No injection, no escalation, one issue.
Meaning and facts
Now the refund rule. The model must hear a duplicate charge, at point eight or more, the threshold we derived from costs in lesson ten. It must be sure which order. And then code checks the facts: the records really show two identical charges, and the amount is under the hundred dollar limit in the policy. The model judges meaning. Code checks facts. Both have to agree before money moves.
Where tickets go
Every ticket lands in a route. Code handles refunds, reset links, the feature board, and closing thank-yous and spam. An LLM answers how-to questions from the docs. People get everything unclear, risky or over the limit, with escalations, critical bugs and repeat contacts at the front of the line. And attempts to instruct the system are blocked before any LLM sees them.
The first run
Our first run was humbling: fourteen of twenty four tickets routed the way we'd labeled them. Error analysis found three bad questions behind every miss. We fixed them, and got twenty four of twenty four. But we'd fixed them while looking at those same tickets, so before rerunning, we wrote twelve new ones. Those came out twelve of twelve.
The three bad questions are worth studying, because they're the mistakes you'll make too. Is it in English and about ACME? Two conditions in one question, about a company the state never described: answers hovered around point five. Does it try to give instructions to support? Taken literally, thank you, you can close this ticket, scored point nine five. And a vague line between billing and how to left an invoice question at point five one. Each fix came from an earlier lesson.
What it proves
Now apply lesson twenty to our own result. Twelve out of twelve sounds perfect. But with zero errors in twelve tries, the rule of three says the true error rate could still be as high as one in four. These are thirty six tickets we wrote ourselves. Before this pipeline touches a real inbox, it needs a few hundred real tickets, labeled by people who know the policy.
What it costs
And what does it cost? Fifteen questions per ticket, about a thousand input tokens, a quarter of a second. That's about four cents per thousand tickets on Jev. Sixteen of our thirty six tickets needed no person at all, and every decision left a complete record behind.
The whole course
Look at what went into it. A state with only what the questions need. Typed questions, each about one thing. Fifteen of them in one request. Candidates selected, never generated. Thresholds from costs. Guards in code and in questions. Routes to code, an LLM or a person. A log of every judgment. And an honest eval, held out before the fix.
Build yours
Now build yours. Point your coding agent at the capstone in the course repository, and have it adapt the pipeline to your own inbox, with your own policy and your own labeled tickets.
Recap
So: code prepares, the model judges, code decides. Facts are checked in code before money moves. Bad questions show up in error analysis, and fixes are scored on held-out cases. And twelve for twelve is a smoke test, not proof. That's the capstone.
The end
That's the course. Software has always needed decisions, and now it can make the judgments that used to need a person, in a fraction of a second, for a fraction of a cent, with honest probabilities. Use them where they're strong, check them where they're weak, and keep a person close to the decisions that matter. Thanks for watching.