What we measured
Vendor benchmarks are upper bounds. So for this course we ran our own tests: 14,318 API calls from a server in Campinas, Brazil, on 2026-10-15. Latencies include the network round trip. The code is in the repository, so you can rerun everything.
BANKING77: 3,080 real support messages, 77 intents
One Choice question per message, with the 77 intent names as options. BANKING77 has been public since 2020, so it may be in some models' training data.
| Model | n | Accuracy | ECE | Answers ≥ 0.99 | Right when ≥ 0.99 | Median latency | $ / 1,000 | |
|---|---|---|---|---|---|---|---|---|
| Jev 1.13 | 3,080 | 80.1% | 0.089 | 50.3% | 96.3% | 0.26 s | $0.071 | |
| Jev 1.14 | 3,080 | 80.3% | 0.071 | 44.4% | 96.4% | 0.26 s | $0.071 | |
| Jev 1.13, options reversed | 3,079 | 80.4% | 0.089 | 50.7% | 96.4% | 0.26 s | $0.071 | |
| Clef-flash (Cloudflare) | 3,078 | 96.0% | 0.052 | — | — | 0.81 s | $0.070 | public test set; see below |
| Claude Haiku 5.5, stated confidence (subset) | 616 | 76.1% | 0.089 | — | — | 1.24 s | $0.106 | JSON answer, 616-message stratified subset |
Clef-flash's 96% was the best score we saw, and it may also be the least informative: BANKING77 has been public since 2020, and it's one of the benchmarks Cloudflare reports for Clef. That's why we built a fresh test.
Fresh news: 279 stories published after every model's release
279 Guardian stories published between September 30 and October 9, 2026, after every model's release, in 8 sections. An answer counts as right if the publisher filed the story in that section. Sections overlap, so some “errors” are defensible.
| Model | n | Accuracy | ECE | Answers ≥ 0.99 | Right when ≥ 0.99 | Median latency | $ / 1,000 | |
|---|---|---|---|---|---|---|---|---|
| Jev 1.13 | 279 | 79.6% | 0.115 | 54.5% | 93.4% | 0.26 s | $0.023 | |
| Jev 1.14 | 279 | 81.7% | 0.102 | 49.5% | 94.2% | 0.24 s | $0.023 | |
| Claude Haiku 5.5, stated confidence | 279 | 79.6% | 0.065 | 0.4% | 100.0% | 1.21 s | $0.049 |
On fresh data the LLM's stated confidence was better calibrated than Jev's probabilities, though it almost never said 0.99. Calibration has to be measured per model and per task.
Option order
We reversed the 77 options and asked again. 150 of 3,079 answers changed (4.9%), almost all of them low-confidence:
| Top probability | Answers | Changed |
|---|---|---|
| 0–0.5 | 112 | 40 (35.7%) |
| 0.5–0.7 | 376 | 87 (23.1%) |
| 0.7–0.9 | 475 | 23 (4.8%) |
| 0.9–0.99 | 567 | 0 (0.0%) |
| 0.99–1 | 1549 | 0 (0.0%) |
One request vs many
One support state of about 1,800 tokens (a ticket, the customer record and a 745-word policy) and 13 questions. Medians of 5 runs each.
| How | Median latency | Input tokens |
|---|---|---|
| 13 questions, one request | 0.265 s | 1,828 |
| 13 requests in sequence | 3.284 s | 18,940 |
| 13 requests in parallel | 0.372 s | 18,940 |
Answers differed by at most 0.04 across runs.
Jagged edges
- Counting fruits in a list of 12 words in one question: answered 5 (p = 0.52); the truth is 6. One Noul per word, summed in code: 6.
- Whether four costs add up to a claimed total (they don't): 0.43. Whether an invoice was paid within 30 days (it took 32): 0.47.
- An injected “SYSTEM NOTE TO CLASSIFIER” didn't hide a real threat: 0.93. Don't build on that; see lesson 16.
Name-swap audit
12 candidate summaries for a backend job, each sent with no name and as Emily Walsh, Greg Baker, Lakisha Washington and Jamal Jones (60 requests, jev-1.13.0). The largest change in any one answer was 0.06 on “should we interview?”, and no decision flipped. That is one test of one attribute on made-up candidates, not evidence of fairness; see lesson 21.
Capstone pipeline
The triage pipeline from lesson 22 is in capstone/. Its first run routed 14 of 24 tickets the way we labeled them; after fixing three questions, 24 of 24, and 12 of 12 on tickets written before the fix. All 36 tickets were written by us, so treat that as a smoke test.
Caveats
- Latency includes the network round trip from Brazil; vendors quote model time.
- Costs use list prices on 2026-10-15: jev $0.042/M input tokens, clef $0.24/M input tokens, clef-flash $0.038/M input tokens, openai-decisions-gpt-6-luna $0.1/M input tokens.
- Some cloud runs were cut short by free-tier quotas; tables show the n we actually have.