The only benchmark that matters is the one you ship. Here is the evaluation loop for decision models: real inputs, careful labels, per-question results, enough examples to trust a difference, and a regression test that pins the model version, with what we measured when Jev went from 1.13 to 1.14.
Key takeaways
- Evaluate on real inputs from your own logs, labeled by people who know the domain; there is no way around labeling some data.
- Thirty examples can reveal your failure modes, but measuring accuracy takes hundreds: at 80% accuracy the margin is about ±14 points with 30 labels and ±4.5 with 300.
- Look per question, not just overall: one weak question can hide inside a good average.
- Test every change, even wording: in one test, more careful wording made agreement and calibration worse.
- Pin the model version; in our tests, Jev 1.14 put 6% fewer answers above 0.99, so the same threshold automated less.
Try it with your agent
Paste this into your coding agent:
For our most important decision-model question, build a regression eval: 1. Sample 300 recent real inputs from logs (stratify so rare classes appear) and export them for expert labeling with a short label guide. 2. Pin the exact model version in one constant. 3. Write a test that reports accuracy with its 95% margin, per-question results, calibration error and coverage at our target, and fails if any drops below the current baseline. 4. Run it in CI on every change to questions, criteria, state or model.
Transcript
Intro
Every number in this course came from someone else's data, or ours. None of them is your data. This lesson is about getting the only numbers that count.
Label some data
Hamel Husain, who consults on AI evals, sees teams resist one step more than any other. There is no getting out of it: you have to label some data. It's the only way to build trust.
The loop
Here's the loop. Sample real inputs from your logs, not examples you made up, and include the rare cases on purpose. Have people who know the domain label them, with a written guide, and where two experts disagree, fix the guide first. Run the model and keep every probability. Then measure, fix the biggest failure, and run it again.
How many labels?
How many labels? Husain finds that around thirty examples is often when you stop discovering new kinds of failure. But thirty is far too few to measure accuracy. At eighty percent, the margin of error with thirty labels is plus or minus fourteen points. With three hundred, four and a half. With a thousand, two and a half.
Per question
Look at each question separately. A Better Stack test compared Jev and Clef on five questions about comments, against a Claude referee. Overall, Jev agreed eighty six percent of the time, Clef eighty three. But Clef won three of the five questions. The whole gap came from one question about tone, where Clef scored forty nine percent. It loves calling things neutral.
Test every change
Test every change, even wording. Emil Lindfors had Jev read Norwegian consultation letters about a salmon tax, first with short draft questions, then with careful ones, full of qualifiers about passing mentions and quotes. The careful version was worse. Agreement went down, and calibration error almost tripled, as answers drifted toward the middle. His advice: ask the way you'd ask a colleague across the desk, and keep the fine print in your code.
Pin the version
And pin the model version. When Jev one point fourteen came out, we reran both of our tests. Accuracy held, and calibration improved. But on the banking test, the share of answers at point nine nine or above fell from fifty to forty four percent. If your code auto-approves above point nine nine, the upgrade quietly sends six percent more of your traffic to people. Better model, different thresholds.
Make it a test
So make your eval a test. Pin the exact model. Keep a few hundred labeled, real inputs in the repository. And fail the build if accuracy, calibration or coverage at your target drops. Run it whenever a question, a criterion, the state or the model changes.
Quick check
Quick check. On thirty labeled examples, model A scores eighty seven percent and model B eighty three. What can you conclude?
Try it
Try it. Have your agent turn one of your decisions into a pinned regression eval.
Recap
So: label some data, from real inputs, with experts. Thirty labels find failures, hundreds measure them. Look per question. Test every change, even wording. And pin the version, because a better model can still move your thresholds. Next: deciding responsibly.