# System One > Learn to build with System One decision models like TypeSafe's Jev, OpenAI's Decisions API, Cloudflare's Clef and AWS's Strands Decider. 22 short animated lessons on typed questions, calibrated probabilities, thresholds, composition patterns and evaluation, distilled from the docs, 57 talks and 70+ articles and papers, plus original measurements. Free and open source, and your agent can take it too. # Lesson 1: Software needs decisions, not essays > Language models write beautifully, but most software needs a decision it can branch on. Here is the prompt-and-parse problem, the new shape of a decision model (state and typed questions in, typed answers and probabilities out), and how a whole category of these models appeared in three weeks. Course: System One (https://learn-system-one.pages.dev/) · Module 1: The idea · 4 min · Watch: https://learn-system-one.pages.dev/lessons/software-needs-decisions/ ## Key takeaways - Most AI calls inside software are really decisions: which team, is it urgent, is this allowed. - Prompt-and-parse (ask an LLM for JSON, then parse and retry) works, but it is slow, fragile and gives you no trustworthy confidence. - A decision model takes state plus typed questions and returns typed answers with probabilities. It cannot return a value you did not offer. - Code keeps control of the workflow; the decision model answers narrow questions; hard cases go to a reasoning model or a person. - In three weeks of September and October 2026, TypeSafe, OpenAI, Cloudflare and AWS all shipped models of this kind. ## Practices for coding agents - When an LLM call exists only to produce a label, a yes/no or a rating that code branches on, consider replacing it with a typed question to a decision model. - Keep control flow, rules and side effects in code; use the model only for the judgment code cannot express. - Never parse free text to recover a decision when a typed answer with probabilities is available. ## Lesson transcript ### Intro (0:01) Language models can write an essay about anything. But most software doesn't need an essay. It needs a decision it can act on. This course is about a new kind of model built for exactly that. ### The gap (0:14) Diogo Almeida helped build the methods behind ChatGPT at OpenAI. When he launched his company, TypeSafe, he opened with one question. Models have been superhuman at chat for years, so where is all the automation? ### Prompt and parse (0:29) Here's how most apps ask a language model for a decision today. Ask for JSON in the prompt. Parse the reply. Then wrap everything in retries, because sometimes the answer gets cut off, or invents a field, or a team you never offered. And the confidence it writes is just more text. ```python reply = llm(f"Classify this ticket. Answer as JSON:" f" team, refund_requested, confidence.\n{ticket}") try: data = json.loads(reply) except json.JSONDecodeError: data = retry() if data["team"] not in TEAMS: data = retry() confidence = float(data["confidence"]) ``` Developer Daniel Sam put the mismatch in two sentences. Software doesn't want sentences. It wants values it can branch on. ### The new shape (0:57) A decision model flips the interface. You send state, which is just your data, and a few typed questions: a yes or no, a choice from your options, a score on your scale. Back comes one typed answer per question, each with probabilities. All of them together, in a quarter of a second. Nothing to parse. Why so fast? A language model writes its answer one token at a time. A decision model reads your data once and scores every allowed answer in parallel. On the same support messages, a small, fast LLM took about one point two seconds per answer. Jev took about a quarter of a second. ### Cheap enough to ask a lot (1:38) It's also cheap enough to ask a lot. For this course, we sent all 3,080 messages from a public banking support dataset through Jev. It took about a hundred seconds, and the whole run cost twenty two cents. ### A category in three weeks (1:52) TypeSafe launched Jev on September fifteenth and called it the first System One model. Two weeks later, OpenAI previewed a Decisions API. Cloudflare released Clef with open weights, and AWS released Strands Decider, small enough to run on a laptop. By the time Jev 1.14 shipped on October fourteenth, decision models were a category. ### Who does what (2:16) This isn't about replacing language models. Your code keeps control of the workflow: the rules, the math, the side effects. The decision model answers narrow questions fast. And when a case is genuinely hard, a reasoning model or a person takes it. A simple test. If the output is one of a known set of answers that code will act on, it's a decision: route this, is it urgent, does this break a rule, which of these candidates matches. If the output is new text, like a reply, a summary or code, you still want a language model. ### Quick check (2:53) **Quiz:** Which job is the best fit for a decision model? A. Draft a polite reply to an angry customer B. Decide which team should handle a ticket C. Summarize a 40-page contract D. Write a SQL query from a question Answer: B. Routing has a known set of answers that code acts on. The others produce new text, which is a language model's job. ### Try it (3:03) Try this with your coding agent. Have it find every place your code asks a language model for a decision and parses the answer. That list is your shortlist for the rest of this course. ```text Find every place in this codebase where we call an LLM and then parse its output into a decision: a label, a yes/no, a score, or one item from a list. For each one, give file:line, the decision it makes, the allowed answers, and whether it could be a typed question (yes/no, choice or score) to a decision model. Don't change any code yet. ``` ### Recap (3:15) So: software needs decisions, not essays. Prompt and parse is slow and fragile. Decision models take state and typed questions and return typed answers with probabilities. Next, where that idea comes from: System One and System Two. ## Sources - Diogo Almeida: Introducing System One Models & Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev - Founders You Should Know: Diogo Almeida - TypeSafe AI (at 0:56). https://www.youtube.com/watch?v=LE3bGTaAgOE&t=56s - a16z: How Jev Turns AI Into Software That Gets Things Done (at 39:59). https://www.youtube.com/watch?v=Ut3LOjKNJaE&t=2399s - Syntax: Jev Explained: Demos and Use Cases (at 3:51). https://www.youtube.com/watch?v=QbYBRjOaGOo&t=231s - Daniel Sam: The Model That Doesn't Talk. https://dev.to/danielsamfdo/the-model-that-doesnt-talk-5adm - TypeSafe AI docs: Introduction. https://docs.typesafe.ai/introduction - TypeSafe AI docs: How to build with TypeSafe. https://docs.typesafe.ai/concepts/how-to-build-with-system-one - OpenAI docs: Decisions (OpenAI API guide). https://developers.openai.com/api/docs/guides/decisions - Cloudflare docs: Introducing Clef: Cloudflare's first open-source decision models, now on Workers AI. https://developers.cloudflare.com/changelog/post/2026-10-01-clef-workers-ai/ - Marc Brooker; Mike Chambers; Fabio Nonato de Paula: Introducing Strands Decider 2B: a small, open source, decision model. https://strandsagents.com/blog/introducing-strands-decider/ - Flavio Copes: How to build typed decisions with Jev. https://flaviocopes.com/jev/ - Simon Willison: Jev introduces a new shape of LLM—System One, aka Decision Models. https://simonwillison.net/2026/Sep/21/jev/ License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 2: System One and System Two > The name comes from Daniel Kahneman: fast, automatic judgment versus slow, effortful reasoning. Here is what the metaphor gets right about decision models, where it breaks, and the rule of thumb it gives you: if an expert could answer in a few seconds, it is a System One question. Course: System One (https://learn-system-one.pages.dev/) · Module 1: The idea · 3 min · Watch: https://learn-system-one.pages.dev/lessons/system-one-and-system-two/ ## Key takeaways - Kahneman's System 1 is fast, automatic and effortless; System 2 is slow, serial and effortful. Decision models aim at the first kind of judgment. - Rule of thumb: if a knowledgeable person could answer in a few seconds with the right context, it is a System One question. - Tasks that need several steps of reasoning, new text or planning are System Two work: send them to a reasoning model, a person, or decompose them. - The useful architecture is a layer cake: code is exact, the decision model judges, the language model thinks. - The two systems are a metaphor, and fast does not have to mean sloppy: what makes fast judgments usable is honest uncertainty. ## Practices for coding agents - Before calling a decision model, check that each question is a few-second judgment for an expert given the state; if not, split it or route it to a reasoning model. - Keep exact computation (math, dates, counting, lookups) in code, never in a fast judgment. - Escalate System Two work (multi-step reasoning, writing, planning) to an LLM or a person instead of forcing it into one question. ## Lesson transcript ### Intro (0:01) Why call a model System One? The name comes from psychology, and it hands you the most useful rule in this course: which questions belong to a fast model, and which don't. ### Two ways to think (0:13) Try this. What's two plus two? The answer just appears. Now, what's seventeen times twenty four? You can feel yourself start to work. Daniel Kahneman, who won a Nobel prize for his work on judgment and decision making, called these System 1 and System 2. In his Nobel lecture he described them like this. System 1 is fast, automatic, effortless and associative. System 2 is slower, serial, effortful and deliberately controlled. ### Fast is not dumb (0:46) And System 1 isn't the dumb one. Kahneman's point is that this is where skill lives. A chess master's strong moves come to mind instantly. System 2 just checks them. ### Models on the spectrum (0:58) Models sit on the same spectrum. A decision model reads the state once and answers in a single pass, in a fraction of a second. Reasoning models do the opposite: they spend tokens thinking, step by step, and trade time for accuracy. ### The rule of thumb (1:15) So here's the rule of thumb, straight from the TypeSafe docs. Make each question a gut check: the kind of judgment a knowledgeable person could make in a few seconds, given the right context. Is this customer asking for a refund? Which team should handle it? How severe is this bug? An expert answers those at a glance. But should we make an exception for this customer, given three years of history? Write the apology? Plan the rollout? Those need System Two. ### The layer cake (1:46) Developer Daniel Sam summed up the architecture in one line: put a decision layer between your rules and your reasoner. Let code be exact, let the model judge, let the LLM think. ### Where the metaphor breaks (1:59) One caution. Kahneman himself called the two systems fictitious characters, a way to talk, not two boxes in the brain. And System 1 thinking is famous for its biases. A fast model can be confidently wrong too. What makes fast judgments usable in software is honest uncertainty, which is module three of this course. ### Quick check (2:22) **Quiz:** Which of these is a System One question? A. Given this contract and our policy history, should we waive the penalty? B. Does this message mention a delivery problem? C. Write a migration plan for this database D. How many invoices in this list are overdue? Answer: B. An expert spots a delivery problem at a glance. Waiving a penalty and planning a migration need reasoning or writing, and counting overdue invoices is exact work for code. ### Try it (2:31) Try it. Take the list of hidden decisions from the last lesson and have your agent sort them into System One questions and System Two work. ```text For each LLM decision we found in this codebase, classify it: - System One: an expert could answer it in a few seconds from the given data - System Two: it needs multi-step reasoning, new text, or planning - Code: it is exact (math, dates, counting, lookups) For every System Two item, propose how to split it into narrow questions plus code, or say why it should stay with an LLM. ``` ### Recap (2:41) So: System One is fast, automatic judgment, and that's what decision models aim for. If an expert could answer in seconds, it's a System One question. Code stays exact, and System Two takes the hard cases. Next: what's actually inside a decision model. ## Sources - kahneman-nobel. https://www.nobelprize.org/uploads/2018/06/kahnemann-lecture.pdf - Lex Fridman: Daniel Kahneman: Thinking Fast and Slow, Deep Learning, and AI | Lex Fridman Podcast #65 (at 9:09, 14:49). https://www.youtube.com/watch?v=UwwBG-MbniY&t=549s - Talks at Google: Thinking, Fast and Slow | Daniel Kahneman | Talks at Google (at 16:40, 20:41). https://www.youtube.com/watch?v=CjVQJdIrDJ0&t=1000s - Preserve Knowledge: Yoshua Bengio | From System 1 Deep Learning to System 2 Deep Learning | NeurIPS 2019 (at 4:00). https://www.youtube.com/watch?v=FtUbMG3rlFs&t=240s - Andrej Karpathy: [1hr Talk] Intro to Large Language Models (at 36:00). https://www.youtube.com/watch?v=zjkBMFhNj_g&t=2160s - TypeSafe AI docs: Introduction. https://docs.typesafe.ai/introduction - Sam Witteveen: Using Jev In Your Agent Harness (at 0:07). https://www.youtube.com/watch?v=zaLQ0AnY9dI&t=7s - Daniel Sam: The Model That Doesn't Talk. https://dev.to/danielsamfdo/the-model-that-doesnt-talk-5adm - Modal: jev: on | Diogo Almeida founder of Typesafe AI, Runtime 2026 (at 6:53). https://www.youtube.com/watch?v=ip7Qej0ONoE&t=413s - Diogo Almeida with swyx: Jev: System One models for Prod, not God (Latent Space podcast) (at 81:23). https://www.youtube.com/watch?v=cFx9Z3ZXca0&t=4883s - Diogo Almeida: Introducing System One Models & Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev - Wikimedia Foundation, Inc.: Dual process theory - Wikipedia. https://en.wikipedia.org/wiki/Dual_process_theory License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 3: How a decision model works > A language model writes its answer one token at a time. A decision model reads your state once and scores every allowed answer in a single pass. Here is the core trick in a few lines of Python, what production models add on top (trained readout heads, shared state, calibration training), and why typed output is not the same as a correct answer. Course: System One (https://learn-system-one.pages.dev/) · Module 1: The idea · 4 min · Watch: https://learn-system-one.pages.dev/lessons/how-decision-models-work/ ## Key takeaways - An LLM's answer costs one sequential pass per output token; a decision model scores all allowed answers in one pass. - The core trick: read the model's probability for each allowed option instead of generating text, then normalize. - Production decision models add a trained readout head, shared state across questions, and training aimed at calibrated probabilities. - Because it only scores the options you supplied, it can never return a value outside your schema. - Typed is not the same as correct: a decision model can still be confidently wrong. That is why probabilities and calibration matter. ## Practices for coding agents - Treat a decision model's output as a probability distribution over your options, not as a fact. - Never assume 'cannot return an invalid value' means 'cannot be wrong'; gate risky actions on probability and test on your data. - Keep the set of allowed answers complete, including a none-of-the-above option when inputs can fall outside it. ## Lesson transcript ### Intro (0:01) What's actually inside a decision model? Less magic than you'd think. In this lesson, you'll see the core trick in a few lines of code, and what the real models add on top. ### Generate vs score (0:13) Start with a normal language model. It writes its answer one token at a time, and every token is another full pass through the network. A fifty token answer is fifty passes, one after another. A decision model never writes. It reads your state once, and scores every allowed answer in a single pass. ### The core trick (0:34) Here's the core trick, adapted from a popular post called Jev in twenty five lines of Python. Put the options in the prompt with letters. Run a small open model once. Then, instead of letting it write, read the probability it gives each letter as the next token, and normalize. ```python labels = ["A", "B", "C"] choices = ["Legitimate", "Spam", "Phishing"] email = "Payroll asks for your password on a fake page." prompt = f"Choose one.\nEmail: {email}\nA. Legitimate\nB. Spam\nC. Phishing" model.eval(model.tokenize(prompt)) logits = model.scores[-1] x = [logits[token_id(l)] for l in labels] p = softmax(x) ``` On a tiny model, that phishing email comes out about eighty nine percent phishing, eight percent spam, and three percent legitimate. A distribution over exactly your options. Nothing to parse, and no way to answer outside the list. ### What real models add (1:07) Real decision models start from that idea and add three layers. A trained readout head, instead of guessing letter tokens. A shared state, read once, so many questions cost almost nothing extra. And training aimed at one thing: probabilities you can trust. AWS's open Strands Decider is a good look inside. They take a two billion parameter model, throw away the part that generates words, and add a tiny pointer head of about a million parameters that compares each option to the question. One forward pass, no generation, no decoding loop. ### Three training goals (1:45) The training goal is where these models differ most from chatbots. Reinforcement learning from human feedback, which made ChatGPT, rewards answers people prefer. Verifiable rewards made reasoning models good at math and code. TypeSafe calls its third path reinforcement learning for calibrated decisions: reward probabilities that match how often the answer turns out right. ### Typed is not correct (2:09) Now, the big claim at launch was that Jev can't hallucinate. Hacker News pushed back fast. As one commenter put it, it can still forward a billing query to the dev department. Even TypeSafe's founder agreed in the thread: these models are probabilistic, so they can be confidently wrong. So what you get is a probability, and the real promise is that the probability is honest. When a decision model says point nine, it should be right about nine times out of ten. Whether that holds on your data is something you'll learn to measure in module three. ### Quick check (2:45) **Quiz:** Why can't a decision model return an answer outside your list? A. It was fine-tuned on your categories B. It only scores the options you supplied and returns a distribution over them C. It regenerates until the output validates D. A regex filters its output Answer: B. The model never writes text. It assigns probability only to the answers you defined, so anything else simply doesn't exist for it. It can still pick the wrong one. ### Try it (2:56) Try it. Ask your coding agent to build the tiny decider from this lesson with any small open model, and compare it with a hosted decision model on twenty of your own examples. ```text Build tiny_decider.py: given a text and a list of options, put the options in a prompt with letters A, B, C..., run a small open GGUF model (e.g. Qwen3-0.6B via llama-cpp-python) once, and return a probability for each option from the next-token logits (softmax over the letters). Then run it on 20 real examples from this project and print the probabilities next to the labels we expect. ``` ### Recap (3:08) So: decision models read once and score every allowed answer in one pass. The trick is reading probabilities instead of writing text. Real models add a trained head, shared state, and calibration training. And typed doesn't mean correct. Next: the four decision models you can use today. ## Sources - Daniel Sam: The Model That Doesn't Talk. https://dev.to/danielsamfdo/the-model-that-doesnt-talk-5adm - Duarte O Carmo: Jev in 25 lines of Python - NobodyWho. https://www.nobodywho.ai/posts/jev-in-25-lines/ - Arcturus Labs: Supercharging LLM Classifications with Logprobs (at 1:00). https://www.youtube.com/watch?v=FN7oaLcIlvk&t=60s - INI Seminar Room 1: Dr. Desi Ivanova | Estimating and Calibrating Uncertainty in LLMs (at 10:44). https://www.youtube.com/watch?v=Kvgy36vIv-0&t=644s - AWS Strands Labs: strands-decider (README). https://github.com/strands-labs/strands-decider - Cloudflare: Introducing Clef: our open-source decision models, and new RL fine-tuning platform. https://blog.cloudflare.com/clef-decision-models/ - Cloudflare: cloudflare/clef model card. https://huggingface.co/cloudflare/clef - Marc Brooker: Strands Decider: Why Not an Encoder?. https://brooker.co.za/blog/2026/10/04/encoders.html - Diogo Almeida: Introducing System One Models & Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev - The TWIML AI Podcast with Sam Charrington: Asking JEV's Creator the Hard Questions About This Hot New AI Model (at 6:23). https://www.youtube.com/watch?v=9Aato-NfjoU&t=383s - AI Engineer: Jev CEO: I made ChatGPT, now I'm building what's next (at 16:16). https://www.youtube.com/watch?v=cJ0EOzey--o&t=976s - Hacker News commenters: Hacker News: Introducing System One Models and Jev. https://news.ycombinator.com/item?id=49717558 - Flavio Copes: How to build typed decisions with Jev. https://flaviocopes.com/jev/ License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 4: Jev, Decisions, Clef, Strands: the landscape > Four decision models you can use today, one request shape they almost share, and what they cost. Then what we measured ourselves: accuracy on a public benchmark and on fresh news published after every model's release, latency from a real server, and why you should trust your own evals over anyone's leaderboard. Course: System One (https://learn-system-one.pages.dev/) · Module 1: The idea · 4 min · Watch: https://learn-system-one.pages.dev/lessons/decision-models-compared/ ## Key takeaways - Jev (TypeSafe, hosted), OpenAI's Decisions API (hosted), Clef (Cloudflare, open weights) and Strands Decider (AWS, open, runs locally) share one shape: state plus typed questions in, probabilities out. - Jev, Clef and Strands speak the same System One request format; OpenAI's is the same idea with different field names. - Pricing is per input token with free output: a thousand decisions typically cost cents. - Public benchmarks can leak into training data and leaderboards move; vendor latency is model time, not your round trip. - Choose on your constraints (data residency, images, latency from your region, price, fine-tuning) and then measure on your own data. ## Practices for coding agents - Hide the vendor behind a small adapter so you can swap decision models and compare them on the same questions. - Pin a versioned model ID in production and log it with every decision. - Measure latency from where your code runs, and accuracy and calibration on your own labeled examples, before choosing a model. ## Lesson transcript ### Intro (0:01) You can use a decision model today from at least four places. They share one shape, but they differ in where they run, what they cost and how fast they answer from where you are. Let's lay them out, and then look at what we measured ourselves. ### The four (0:16) Jev, from TypeSafe, is a hosted API at four point two cents per million input tokens. OpenAI's Decisions API runs on GPT-6 Luna at ten cents. Cloudflare's Clef comes in two sizes on Workers AI, and you can download the weights. And AWS's Strands Decider is a two billion parameter model you run yourself, even on a laptop. All four charge nothing for output. ### One shape (0:43) And they speak nearly the same language. Jev's request format became a de facto standard: Clef and Strands accept it as is, so switching is a change of endpoint. OpenAI uses the same idea with different names: input instead of state, predicate instead of noul. ### What we measured (1:01) We ran all 3,080 messages of BANKING77, a public dataset of bank support questions with seventy seven intents, through each model we could reach. Jev 1.13 and 1.14 both got about eighty percent right. A fast LLM asked for its answer as JSON got seventy six. And Clef flash got ninety six percent. Before you crown a winner, notice something. BANKING77 has been public since 2020, and it's one of the benchmarks Cloudflare reports for Clef. Public test sets can end up in training data. So we built a fresh test: 279 news stories published after every one of these models was released. There, Jev 1.14 got eighty two percent, and the LLM eighty. ### Latency is local (1:51) Latency tells a similar story. Vendors quote model time, often under a tenth of a second. From our server in Brazil, the full round trip told the truth: Jev answered in a quarter of a second, Clef flash in eight tenths, and the LLM in one point two. ### Cents per thousand (2:09) And the bill? With seventy seven options in every question, about seventeen hundred tokens each, a thousand decisions cost about seven cents on Jev or on Clef flash. Even the LLM was only about eleven cents. At these prices, the question is never can we afford to ask. It's what else should we ask. ### How to choose (2:30) So how do you choose? Start from your constraints. If data can't leave your servers, run open weights: Strands Decider, or Clef yourself. If you need images, OpenAI and Clef read them today. Pick the region closest to your code. Compare the price per thousand of your decisions, not per token. And if you'll want to fine tune, Cloudflare offers that for Clef. And whatever you pick, take TypeSafe's own advice from their essay on benchmarks: run your own private evals, treat public ones with a grain of salt, and don't amplify every number or plot you see. That includes ours. ### Quick check (3:08) **Quiz:** Your data can't leave your servers. Which option fits? A. Jev's hosted API with long context turned on B. OpenAI's Decisions API C. Strands Decider, or Clef's open weights, run on your hardware D. Whichever topped the latest leaderboard Answer: C. Strands Decider and Clef publish their weights, so you can run them where your data lives. Then measure them on your own data. ### Try it (3:20) Try it. Have your agent write one small adapter that can call any of these models with the same questions, and log the model, latency and probabilities, so every later lesson can compare them on your data. ```text Write decide.py with one function: decide(state, questions, provider). questions use the System One shape (noul / choice / score). Support: - jev: POST https://api.typesafe.ai/v1/systemone (TYPESAFE_API_KEY) - clef: Cloudflare Workers AI @cf/cloudflare/clef-flash (same body) - openai: POST /v1/decisions (map state->input, noul->predicate, criteria->choices/levels; model gpt-6-luna) Return the same shape for all. Log provider, model id, latency and probabilities to decisions.jsonl. Add a test that runs one question through every provider that has a key set. ``` ### Recap (3:33) So: four decision models, one shape, cents per thousand decisions. Public benchmarks are upper bounds, latency is local, and the only score that counts is the one on your data. That's module one. Next, how to ask good questions, starting with the simplest: the Noul. ## Sources - TypeSafe AI docs: Models. https://docs.typesafe.ai/models - OpenAI docs: Decisions (OpenAI API guide). https://developers.openai.com/api/docs/guides/decisions - Cloudflare docs: Introducing Clef: Cloudflare's first open-source decision models, now on Workers AI. https://developers.cloudflare.com/changelog/post/2026-10-01-clef-workers-ai/ - Cloudflare: Introducing Clef: our open-source decision models, and new RL fine-tuning platform. https://blog.cloudflare.com/clef-decision-models/ - Cloudflare: cloudflare/clef model card. https://huggingface.co/cloudflare/clef - AWS Strands Labs: strands-decider (README). https://github.com/strands-labs/strands-decider - The New Stack: AWS launches a local answer to TypeSafe's Jev decision model. https://thenewstack.io/aws-strands-decider-model/ - Frederic Lardinois: OpenAI answers TypeSafe's Jev with a Decision API built on Luna. https://thenewstack.io/openai-decision-api-luna/ - jaredpalmer/kev: jaredpalmer/kev: Jev-like family of decision models built on top of Qwen3.5/3.8 you can train and run on…. https://github.com/jaredpalmer/kev - JevBench authors: JevBench v1 (benchmark for Jev-class typed decision models). https://github.com/doxanocap/jevbench - Benchmark Heaven: Jev-class models — JevBench v1.0. https://benchmarkheaven.com/jev-models/v1 - TypeSafe AI: Lies, Damned Lies, and Benchmarks. https://typesafe.ai/blog/antibenchmaxxing - Flavio Copes: How to build typed decisions with Jev. https://flaviocopes.com/jev/ - Mark Kashef: OpenAI's Decisions API just dropped. Here's how it compares to Jev. (at 9:12). https://www.youtube.com/watch?v=uTU5Ihgl_7Q&t=552s License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 5: Noul: the probability of yes > The simplest decision is yes or no. A Noul returns the probability that the answer is yes, and that single number carries both the answer and the doubt. Here is how to read it, where to put thresholds, why 0.5 never means medium, and how to phrase yes/no questions the model can't misread. Course: System One (https://learn-system-one.pages.dev/) · Module 2: Asking good questions · 4 min · Watch: https://learn-system-one.pages.dev/lessons/noul-yes-no-questions/ ## Key takeaways - A Noul returns one number: the probability that the answer is yes. Near 1 is a strong yes, near 0 a strong no, near 0.5 means it can't tell. - 0.5 is not 'medium'. If you need a degree, use a Score with described levels. - Thresholds live in your code and depend on the cost of each mistake; values in the middle can go to a person. - One condition per question, phrased so a high value means yes, with the exact condition spelled out. - Short, plain questions beat long qualified ones; keep the fine print in your code. ## Practices for coding agents - Use a Noul for a clean yes/no condition; use a Score when the answer is a degree. - Write one condition per Noul and phrase it so a high probability means yes; avoid negations. - Threshold Noul values in code, with a review band in the middle sized by the cost of errors. - Add true/false criteria when the boundary is subtle, and keep questions short and literal. ## Lesson transcript ### Intro (0:01) The simplest decision is yes or no. In Jev it's called a Noul, short for Bernoulli. You ask a yes or no question, and you get back one number. Let's learn to read it, and to write questions it can't misread. ### Reading a Noul (0:16) Here's one question, is the customer asking for a human agent, asked about six messages. Thanks, that fixed it: point oh three, a strong no. I need this sorted today: point two six, urgent but not asking for a person. Are you a bot: point three six, a hint without a request. And the last one is a clear yes at point nine nine. So the number is the answer and the certainty at once. That's why a Noul has no separate confidence field. And it means point five is not medium. Point five means the model gives yes and no the same chance. It can't tell. ### Degree needs a Score (0:57) If you actually want a degree, ask for one. Is the candidate strong in Python climbs from point oh three to point nine two, but the spacing between candidates means nothing you chose. A Score with four described levels puts each candidate on your scale: no experience, some familiarity, regular use, deep expertise. ### Thresholds live in code (1:19) In code, you turn the probability into behavior with thresholds. A clear yes goes one way, a clear no the other, and anything in the middle goes to a person. The thresholds are yours, and they're one line to change. ```python p = answers["is_human_escalation"].noul YES, NO = 0.8, 0.2 # yours to tune if p >= YES: route_to_agent(ticket) # clear yes elif p <= NO: route_to_bot(ticket) # clear no else: send_to_review(ticket) # can't tell ``` Where should the thresholds go? It depends on what a mistake costs. If acting on a false yes is expensive, like issuing a refund or paging someone at night, raise the bar. If missing a true yes is expensive, like a safety report, lower it. And the review band in the middle is your budget for human attention. ### Writing a good Noul (1:55) Now, writing them. First, one condition per question. Is the customer angry and asking for a refund came back point oh eight for this furious customer. Split it, and you learn more: angry, point nine eight. Refund, point oh four. They want a fix, not their money back. Three more rules. Phrase it so a high value means yes: does it contain personal data, not is it free of personal data. Your code will read it backwards someday. Avoid negations: AWS found that questions like which does this not fit often get answered as if the not weren't there. And say exactly what you mean: does the user want a human now, if now is what matters. ### Criteria for subtle boundaries (2:43) When the boundary is subtle, add criteria: a short description of what counts as yes and what counts as no. Here, has the customer contacted us before scores point eight six on its own. With criteria that name prior attempts and tickets, it's a firmer point nine four. ### Keep it short (3:00) And keep it short. Emil Lindfors tested this in early access. Rewriting his questions into long, carefully qualified instructions lowered agreement and tripled the calibration error, pushing answers toward the middle. His advice: write them the way you'd ask a colleague across the desk. ### Quick check (3:20) **Quiz:** “Is this bug severe?” returns ≈0.5 on most tickets. Best fix? A. Lower the threshold to 0.5 B. Average it over three calls C. Ask about an exact condition, or use a Score with described levels D. Add more context to the state Answer: C. “Severe” has no boundary, so the model can't tell. Ask a checkable condition (“Does it block checkout?”) or measure degree with a Score. ### Try it (3:33) Try it. Have your agent turn the yes or no decisions in your code into Nouls, with exact conditions, and thresholds in code. ```text Find yes/no decisions we currently ask an LLM to make. For each, write a Noul question: one condition, phrased so a high probability means yes, no negations, the exact condition spelled out. Add true/false criteria only where the boundary is subtle. Then add YES/NO thresholds in code, with a middle band that goes to human review, and explain each threshold by the cost of a false yes versus a missed yes. ``` ### Recap (3:42) So: a Noul is the probability of yes. Point five means can't tell, not medium. Thresholds live in code and follow the cost of mistakes. And one exact condition per question, phrased so high means yes. Next: when the answer is one of several options, the Choice. ## Sources - TypeSafe AI docs: Noul. https://docs.typesafe.ai/primitives/noul - TypeSafe AI docs: Primitives (Questions). https://docs.typesafe.ai/primitives - Simon Willison: Jev introduces a new shape of LLM—System One, aka Decision Models. https://simonwillison.net/2026/Sep/21/jev/ - Flavio Copes: How to build typed decisions with Jev. https://flaviocopes.com/jev/ - Unsiloed AI: Document Review With Unsiloed and TypeSafe Jev. https://www.unsiloed.ai/blog/document-review-unsiloed-typesafe-jev - Emil Lindfors: An early-access test of TypeSafe's Jev: calibrated judgments for half a cent. https://lindfors.no/blog/a-first-look-at-typesafes-jev/ - AWS Strands Labs: strands-decider: evaluation. https://github.com/strands-labs/strands-decider/blob/main/evaluation/README.md - Hacker News commenters: Hacker News: Introducing System One Models and Jev. https://news.ycombinator.com/item?id=49717558 - Hamel Husain: Using LLM-as-a-Judge For Evaluation: A Complete Guide – Hamel’s Blog. https://hamel.dev/blog/posts/llm-judge/ - Hamel Husain: LLM Evals: Common Mistakes (at 12:48). https://www.youtube.com/watch?v=GL0XhAj5LPE&t=768s - IndyDevDan: 10 Levels of Jev For Agentic Engineers (at 2:20). https://www.youtube.com/watch?v=_U-O5lYhJ7Q&t=140s License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 6: Choice: one answer from a set > A Choice picks one option from a set you define and returns a probability for every option. Here is how to read the distribution and its confidence, why a none-of-the-above option matters, how contrastive descriptions sharpen close calls, and what reversing the option order did to 3,080 real answers. Course: System One (https://learn-system-one.pages.dev/) · Module 2: Asking good questions · 4 min · Watch: https://learn-system-one.pages.dev/lessons/choice-questions/ ## Key takeaways - A Choice returns the chosen option, a probability for every option, and a confidence that summarizes how peaked that distribution is. - A split distribution is information: a ticket that belongs to two teams should look split. - Always offer a none-of-the-above option when inputs can fall outside your list; otherwise the model must pick something. - Option names and descriptions are both sent to the model; for close calls, describe what each option covers and what it doesn't. - Option order can sway uncertain answers: in our test, reversing 77 options changed 4.9% of answers, none above 0.9 probability. ## Practices for coding agents - Give every Choice the full list of options with short descriptions, plus an explicit other / none option when coverage isn't guaranteed. - Use contrastive descriptions (what it covers, what it is not for, examples) for options that get confused. - Treat low-confidence Choice answers as candidates for review, and re-check close calls with the options reordered. ## Lesson transcript ### Intro (0:01) Most decisions in software pick one thing from a list: which team, which category, which intent. That's a Choice. Let's see what it gives back, and the four habits that make it reliable. ### Reading a Choice (0:14) Here's a messy ticket. Shoes arrived late and in the wrong size, and there are two charges on the card. Which team should handle it? Returns wins at point six three, but billing gets point three three, because of the double charge. That split is honest. This ticket belongs to two teams. Confidence turns that shape into one number. Zero means the probability is spread evenly. One means it's all on one option. Here it's point four five. It's computed from the probabilities, so you can always check it, or use your own measure, like the gap between the top two options. ### None of the above (0:52) Habit one: always offer a way out. Someone writes in to apply for a designer job. Your options are returns, shipping and billing. Without an other option, the model has to pick something: billing, at point five, spread thin. Add other, described as none of the above, and it goes there with all the probability. ### Describe the options (1:14) Habit two: describe the options. The model reads both the option names and their descriptions. For options that get confused, say what each one covers, what it's not for, and give an example or two. On a borderline question about where a disposable card works, plain names gave point nine. Contrastive descriptions gave a clean one point oh. In code, it looks like this. Each option gets the same fields, so the model can compare them side by side. You only need this for the pairs that actually get confused. Clear options can keep a one line description, or none at all. ```python "topic": Choice( instructions="Which disposable virtual card topic is the user asking about?", criteria={ "get_card": {"what": "Purpose, eligibility, or setup", "not_for": "Quantity, transaction, or merchant limits", "examples": ["What are disposable cards for?"]}, "card_limits": {"what": "Quantity, transaction, or merchant limits", "not_for": "Purpose, eligibility, or setup", "examples": ["Where can I use a disposable card?"]}, }) ``` ### Give the full list (1:52) Habit three: give the full list. Jev accepts up to two hundred fifty five options, and each one costs only a few tokens. Don't hand-pick a shortlist; offer every team, every category. For deep taxonomies with thousands of leaves, go level by level, which we'll cover in the composition module. ### Option order (2:14) Habit four: watch the order. TypeSafe's docs warn that Jev can lean toward the first option. So we reversed the order of all seventy seven intents and reran 3,080 messages. Four point nine percent of answers changed. But look where: thirty six percent of answers under point five flipped, and not a single answer above point nine. That's the pattern you'll see all through this course. When the model is sure, small changes don't move it. When it isn't, they do, and the probability tells you which case you're in. So route low confidence answers to review, and for close calls you care about, ask again with the options shuffled. ### Quick check (2:56) **Quiz:** A message fits none of your five departments, and there's no “other” option. What happens? A. The API returns an error B. The model returns an empty answer C. The model must spread probability over your five options and pick one D. The model invents a sixth department Answer: C. A Choice only scores the options you give it, so it has to pick one, usually with a flat, low-confidence distribution. Add “other” so it has somewhere honest to go. ### Try it (3:12) Try it. Have your agent audit every Choice in your code for the four habits. ```text Review every multiple-choice decision we ask a model to make. For each: 1. Is the option list complete? Add an explicit "other" / none option if inputs can fall outside it. 2. Does each option have a one-line description? For options that are often confused, add contrastive what / not_for / examples fields. 3. Does code treat low-confidence answers differently? 4. Add a test that re-asks 20 known close calls with the options shuffled and reports any answer that changes. ``` ### Recap (3:18) So: a Choice gives you a distribution, and its confidence says how peaked it is. Offer a way out. Describe the options that get confused. Give the full list. And remember that low confidence answers are the fragile ones. Next: the Score. ## Sources - TypeSafe AI docs: Choice. https://docs.typesafe.ai/primitives/choice - TypeSafe AI docs: Confidence. https://docs.typesafe.ai/confidence - TypeSafe AI docs: How to build with TypeSafe. https://docs.typesafe.ai/concepts/how-to-build-with-system-one - TypeSafe AI docs: Jev 1.13 jaggedness. https://docs.typesafe.ai/model-jaggedness/jev-1.13 - AWS Strands Labs: strands-decider (README). https://github.com/strands-labs/strands-decider - Amir Rafe amir rafe edu: Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with…. https://arxiv.org/html/2609.24052 - Pezeshkpour, Hruschka: Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions. https://arxiv.org/abs/2308.11483 - Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. https://arxiv.org/abs/2306.05685 - Daniel Sam: The Model That Doesn't Talk. https://dev.to/danielsamfdo/the-model-that-doesnt-talk-5adm License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 7: Score: a position on a scale you describe > A Score places the state on ordered levels you describe and returns a position that can fall between them. Here is how the number is computed, why levels must describe situations rather than degrees, why one Score should measure one thing, and how an extra top level separates the dangerous cases. Course: System One (https://learn-system-one.pages.dev/) · Module 2: Asking good questions · 4 min · Watch: https://learn-system-one.pages.dev/lessons/score-questions/ ## Key takeaways - A Score returns a position along your levels: each level number times its probability, added up. It can land between levels. - Describe situations, not degrees: 'workaround exists' gives the model something to match; '1' or 'moderate' doesn't. - The same score can hide different shapes; read the probabilities and confidence, not just the number. - One dimension per Score; split compound scales and combine the parts in code. - Give a rare extreme case its own top level so it doesn't hide inside 'very bad'. ## Practices for coding agents - Write every Score level as a concrete, self-contained situation; never use bare numbers or relative words like 'worse'. - Keep each Score to one dimension; normalize scores (divide by the top level) before weighting them together. - Add a separate top level for rare extremes that need different handling (abuse, outages, safety). - Use the score to rank or threshold, not to reconstruct an exact quantity between levels. ## Lesson transcript ### Intro (0:01) Some answers aren't yes or no, or one of a few categories. They're a position on a scale: how severe, how frustrated, how much experience. That's a Score. ### Reading a Score (0:13) Three levels for bug severity: cosmetic, broken but there's a workaround, and blocking with no workaround. This bug crashes the page in Safari, but works in Chrome, and some customers only use Safari. The model puts point five six on workaround and point four four on blocking, so the position lands at one point four four. The math is simple. Each level number, times its probability, added up. Zero times zero, plus one times point five six, plus two times point four four. One point four four. And confidence is low, point three four, because the model is torn between two levels. ### Describe situations (0:56) Rule one: describe situations, not degrees. Each level is judged on its own, so a level called just two, or moderately severe, gives the model nothing to match. For a pixel-misaligned button, levels written as bare numbers scored point seven one, unsure. With described levels: zero point zero, at full confidence. ### One dimension (1:21) Rule two: one dimension per Score. Rate a contractor on a scale from late, unskilled and rude to punctual, skilled and friendly. One who arrived on time, couldn't answer basic questions, and was very polite gets a confident Mixed. Which tells you nothing. Split it into three Scores, and you get on time, no skill, very polite. This isn't just a model quirk. Decision researcher Olivier Sibony gives the same advice to human hiring panels: evaluate each criterion separately from the others, so one strong impression doesn't color the rest. Then combine them, which, for us, happens in code. ### Combine in code (2:04) Before you combine Scores, normalize them. A four level scale runs from zero to three, a three level scale from zero to two. Divide each by its top level, so everything runs from zero to one. Now your weights mean what they say. ```python a = response.answers severity = a["severity"].score / 2 # 3 levels: 0..2 frustration = a["frustration"].score / 2 # 3 levels: 0..2 quality = a["report_quality"].score / 3 # 4 levels: 0..3 priority = 0.6 * severity + 0.3 * frustration + 0.1 * quality ``` ### Room for the extreme (2:20) Rule three: give the rare extreme its own level. On a calm, annoyed, very angry scale, a customer threatening your support team scores one point eight eight, right next to someone who's merely furious. Add a fourth level, abusive or threatening, and the threat lands at two point eight six, while ordinary anger stays at two. ### What a score is not (2:44) And one warning from the Jev docs. A score is a position for ranking and thresholds, not a measurement. Don't interpolate it into an exact quantity. One point four four doesn't mean forty four percent of customers lack a workaround. ### Quick check (2:59) **Quiz:** A 1-to-5 “reply quality” Score keeps landing near 3 with low confidence. Best fix? A. Add more levels, say 1 to 10 B. Describe each level as a concrete situation, and split quality into separate dimensions C. Average three runs D. Switch to a Noul: “Is this reply good?” Answer: B. Bare numbers give the model nothing to match, and “quality” bundles accuracy, tone and completeness. Describe situations and score each dimension separately. ### Try it (3:17) Try it. Have your agent rewrite one rating in your product as well-built Scores. ```text Find one place where we rate something on a scale (priority, quality, severity, risk). Rewrite it as Score questions: - one dimension per Score; split compound scales - every level a concrete, self-contained situation (no bare numbers) - a separate top level for rare extremes that need different handling Normalize each score by its top level, combine with explicit weights in code, and show 5 real examples with each score and the combined result. ``` ### Recap (3:23) So: a Score is a position, level times probability, added up. Describe situations, not degrees. One dimension per Score. Give the extreme its own level. And use the number to rank, not to measure. Next: the state, what you send the model to judge. ## Sources - TypeSafe AI docs: Score. https://docs.typesafe.ai/primitives/score - TypeSafe AI docs: Confidence. https://docs.typesafe.ai/confidence - TypeSafe AI docs: Jev 1.13 jaggedness. https://docs.typesafe.ai/model-jaggedness/jev-1.13 - OpenAI docs: Decisions (OpenAI API guide). https://developers.openai.com/api/docs/guides/decisions - Flavio Copes: How to build typed decisions with Jev. https://flaviocopes.com/jev/ - Hamel Husain: LLM Evals: Common Mistakes (at 13:46). https://www.youtube.com/watch?v=GL0XhAj5LPE&t=826s - Hamel Husain: Using LLM-as-a-Judge For Evaluation: A Complete Guide – Hamel’s Blog. https://hamel.dev/blog/posts/llm-judge/ - CFA Institute: Olivier Sibony: How can you overcome noise and bias in decision making? (at 14:50). https://www.youtube.com/watch?v=wiYaXOGE_e4&t=890s License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 8: State: give it what it needs > The state is everything the model sees besides your questions. Here is how to shape it (named fields, paths the questions point at, the record next to the policy), what we measured when we buried the one thing that mattered, and the limits on length, language and context you should design around. Course: System One (https://learn-system-one.pages.dev/) · Module 2: Asking good questions · 4 min · Watch: https://learn-system-one.pages.dev/lessons/designing-state/ ## Key takeaways - State is the material you would hand an expert panel before asking: the message, the record, the policy. - Use a JSON object with named fields, and point each question at the field it is about with a backticked path. - Unrelated text in its own named field barely hurt in our test; asking about one item buried by index among 40 similar ones dropped accuracy from 98% to 72%. - Name the thing you are asking about: one item per request, or put the record in the question itself. - Filter in code before you send; long context exists but is slower and less accurate, and English is still the strongest language. ## Practices for coding agents - Send state as a JSON object with descriptive field names; reference fields in questions with backticked paths like ticket.messages[0].text. - Put the decision's evidence (policy, record, history) in the state, and only what the questions need. - Never make the model find one item by index in a long list of similar items; send that item on its own or inside the question. - Write the complete question in instructions; question IDs are not sent to the model. ## Lesson transcript ### Intro (0:01) Every request has two halves: your questions, and the state they're asked about. Most failures we'll see in this course start on the state side. So let's shape it well. ### What state is (0:12) The TypeSafe docs have a good image for it. Think of the state as the material you'd present to a panel of experts before asking them to make a judgment. The customer's message. Their order. The policy that applies. Everything an expert would need, and nothing they wouldn't. ### Shapes of state (0:30) State can be a plain string, which is fine for one message. An array works for a sequence, like a conversation. But for most real decisions, use an object, so every part has a name and the relationships stay clear. ### Point at the field (0:45) Then point each question at the field it's about, with a path in backticks. Does the policy support the refund requested in the ticket, given the order's charges? The model knows exactly which parts to compare. ### Bury it, lose it (0:58) How much does extra material hurt? We tested sixty banking messages, half of them about refunds, with one Noul: does the ticket ask about a refund? Alone, ninety eight percent right. With six thousand tokens of unrelated handbook in its own field: still ninety eight. But bury the same ticket among forty similar tickets and ask about ticket number seventeen, and it drops to seventy two, missing seventeen of the thirty refund requests. So the lesson isn't just keep the state short. It's name the thing you're asking about. Don't make the model hunt for item seventeen in a list of lookalikes. Send that item on its own, or put it right in the question. ### Data in the question (1:43) Questions can carry data too. To check a new resume against candidate records, put each record inside its own question's instructions, next to the same question text. Twenty records, twenty Nouls, one request, and each answer is about exactly one record. ```python SAME = "Is the resume for the same person as `potential_duplicate`?" questions = { f"same_as_{c['id']}": Noul(instructions={ "potential_duplicate": {"name": c["name"], "location": c["location"]}, "question": SAME, }) for c in candidates # one question per record } client.system_one(state={"resume": resume}, questions=questions) ``` ### Limits to design around (2:01) A few limits to design around. Question IDs aren't sent to the model, so write the full question in the instructions. Jev reads up to sixty four thousand tokens per request, and thirty two thousand for the state plus any one question. Long context can be turned on, but TypeSafe warns it's slower and less accurate. And English is still the strongest language, so test others before you rely on them. ### Facts vs judgments (2:27) One last habit, from TypeSafe's own guide for coding agents. When you store answers and feed them into later requests, keep what you observed separate from what the model inferred. A field that says refund requested, point nine nine, is a judgment, not a fact. Label it, and check it's still fresh before you act on it. ### Quick check (2:49) **Quiz:** Is ticket 17 of 40 asking for a refund? Best state? A. All 40 tickets in an array; ask about tickets[17] B. All 40 tickets plus the refund policy, for context C. Ticket 17 alone, as a named field; ask about that field D. A summary of all 40 tickets Answer: C. In our test, asking about one ticket by index among 40 lookalikes missed 17 of 30 refunds. Send the item you're asking about on its own, under a clear name. ### Try it (3:02) Try it. Have your agent design the state for one decision in your code, and test what happens when you bury it. ```text Pick one decision we make with a model. Design its state as a JSON object with named fields: only what an expert would need (the message, the record, the relevant policy). Rewrite each question to point at its field with a backticked path. Then run 20 labeled examples three ways: the clean state, the state plus a large unrelated document in its own field, and the item hidden in a list of 20 similar items. Report the accuracy of each. ``` ### Recap (3:10) So: state is what you'd hand an expert panel. Use named fields and point questions at them. Name the thing you're asking about, and never make the model hunt for it. Design around the limits, and keep judgments separate from facts. That's module two. Next: probabilities you can act on, starting with calibration. ## Sources - TypeSafe AI docs: State. https://docs.typesafe.ai/concepts/state - TypeSafe AI docs: Primitives (Questions). https://docs.typesafe.ai/primitives - TypeSafe AI docs: How to build with TypeSafe. https://docs.typesafe.ai/concepts/how-to-build-with-system-one - TypeSafe AI docs: Jev 1.13 jaggedness. https://docs.typesafe.ai/model-jaggedness/jev-1.13 - TypeSafe AI docs: Models. https://docs.typesafe.ai/models - TypeSafe AI docs: Noul. https://docs.typesafe.ai/primitives/noul - TypeSafe AI: TypeSafe agent skill (SKILL.md). https://github.com/typesafe-ai/skills/blob/main/skills/typesafe-ai/SKILL.md - Diogo Almeida with swyx: Jev: System One models for Prod, not God (Latent Space podcast). https://www.latent.space/p/jev - Ki-jung Kim (akaslany): Why We Made Jev — Diogo Almeida, TypeSafe Co-founder & CEO (at 6:29). https://www.youtube.com/watch?v=RW_0PqUFq3w&t=389s License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 9: Calibration: does 0.9 mean 90%? > A probability is only useful if it means what it says. Here is what calibration is, how to see it in a reliability diagram, why chat-trained models drift into overconfidence, what independent tests and our own measurements found for decision models, and how a few hundred labels let you measure and fix it. Course: System One (https://learn-system-one.pages.dev/) · Module 3: Probabilities you can act on · 4 min · Watch: https://learn-system-one.pages.dev/lessons/calibration/ ## Key takeaways - Calibrated means: of all the answers given probability 0.8, about 80% turn out right. - A reliability diagram plots stated probability against how often the model was right; the diagonal is perfect. - Post-training for chat tends to make models overconfident; decision models are trained to keep probabilities honest, but that is a claim to test. - Calibration is a property of each model on each task: in our tests Jev was slightly overconfident at the top, and on fresh news an LLM's stated confidence was better calibrated. - Measure it on a few hundred of your own labels (ECE, Brier score), and fix systematic bias with simple recalibration such as Platt scaling. ## Practices for coding agents - Before relying on probabilities, measure calibration on a labeled sample from your own traffic: a reliability table, ECE and Brier score. - Never assume a vendor's calibration claim holds on your data; re-check after every model version change. - If probabilities are systematically off, fit a recalibration map (Platt scaling or isotonic regression) on held-out labels; it fixes calibration but cannot add information. - Prefer native probabilities to numbers a model writes as text, but measure both if you use both. ## Lesson transcript ### Intro (0:01) Every answer from a decision model comes with probabilities. But a probability is only useful if it means what it says. This lesson is about checking that. ### What calibration means (0:12) Think of a weather forecaster. On the days they say seventy percent chance of rain, it should rain on about seventy percent of them. That's calibration. It's not about any single day. It's about the group. Of all the answers a model gives at point eight, about eight in ten should be right. ### Seeing it (0:30) You see calibration in a reliability diagram. Group answers by the probability the model gave, and plot how often each group was right. Perfect calibration is the diagonal. Here's Jev 1.13 on our 3,080 banking messages. Mostly close, but slightly below the line at the top: answers it gave about point nine nine were right ninety two percent of the time. A little overconfident. ### Why chatbots drift (0:57) Why does this need saying at all? Because chat models are famously badly calibrated. Pretrained models are fairly well calibrated, but post-training for chat, as researcher Desi Ivanova puts it, destroys calibration. People like answers that sound sure, so models learn to sound sure. ### Independent tests (1:17) Independent testers have checked the decision models directly. Ryan Porter ran 3,600 hard reasoning questions through OpenAI's Decisions API and Jev. Luna said ninety nine percent or more on 2,672 answers, and only sixty eight percent of those were right. Jev said ninety nine percent on 1,667 answers, and was right on ninety eight point nine. ### Model by model (1:44) But don't turn that into a rule about paradigms. A team that checked Jev against 2,416 human judgments of police crash reports found it wasn't overconfident there at all. It hedged, staying too close to the middle. Their conclusion is the one to remember: calibration turns out to be a property of the individual model, rather than of the paradigm. We saw the same thing on our fresh news test. Jev 1.14 was more accurate, but its top answers were a bit overconfident. The LLM, asked to write its confidence as a number, was actually better calibrated on this task. Calibration has to be measured, task by task. ### Measuring it (2:28) Two numbers summarize it. Expected calibration error is the average gap between stated probability and actual accuracy, weighted by how many answers fall in each bin. Zero is perfect. The Brier score is the mean squared error of the probabilities themselves. Always guessing fifty fifty on a yes or no question scores point two five, so you want to be well below that. You can measure it in a dozen lines. Bin the answers by stated probability, compare each bin's average probability with its accuracy, and weight the gaps. Run it on a few hundred of your own labeled examples, per question, and again whenever the model version changes. ```python def ece(rows, bins=10): # rows: (stated_probability, was_right) groups = [[] for _ in range(bins)] for p, ok in rows: groups[min(int(p * bins), bins - 1)].append((p, ok)) err = 0.0 for grp in groups: if grp: conf = sum(p for p, _ in grp) / len(grp) acc = sum(ok for _, ok in grp) / len(grp) err += len(grp) / len(rows) * abs(conf - acc) return err brier = sum((p - ok) ** 2 for p, ok in rows) / len(rows) ``` ### Fixing it (3:10) And if it's off in a consistent direction, you can fix it cheaply. Fit a tiny logistic regression from the model's probability to what actually happened, called Platt scaling, on held-out labels. The crash narrative team cut calibration error by a factor of three point three this way. One limit: remapping fixes calibration, but it can't add information the probabilities don't carry. ### Quick check (3:35) **Quiz:** 500 answers stated at 0.9; 400 were right. What does that tell you? A. The model is well calibrated B. It's overconfident at 0.9: those answers are right about 80% of the time C. It's underconfident D. Nothing: calibration is about single answers Answer: B. 400 of 500 is 80%, below the 90% it claimed. Recalibrate, or treat its 0.9 as 0.8 when you set thresholds. ### Try it (3:49) Try it. Have your agent build a calibration report for one of your decisions. ```text For one decision we make with a model, collect 200-300 examples with trusted labels from real traffic. Run the model on all of them and log the probability of the chosen answer. Then report: accuracy, a 10-bin reliability table (stated vs actual), ECE and Brier score. Fit Platt scaling on half the data and report ECE on the other half, before and after. Save the script so we can rerun it on every model version change. ``` ### Recap (3:54) So: calibrated means point eight is right eight times in ten. See it in a reliability diagram, summarize it with ECE and Brier. Chat training drifts toward overconfidence, but every model has to be measured on every task. And a simple recalibration fixes the numbers. Next: turning probabilities into actions with thresholds. ## Sources - ritvikmath: Probability Calibration : Data Science Concepts (at 0:56). https://www.youtube.com/watch?v=AunotauS5yI&t=56s - IBM Technology: What Is Jev? The AI Model That Doesn't Generate Text (at 10:15). https://www.youtube.com/watch?v=YGgNBcIgI4s&t=615s - INI Seminar Room 1: Dr. Desi Ivanova | Estimating and Calibrating Uncertainty in LLMs (at 19:35). https://www.youtube.com/watch?v=Kvgy36vIv-0&t=1175s - Talks at Google: Thinking, Fast and Slow | Daniel Kahneman | Talks at Google (at 45:35). https://www.youtube.com/watch?v=CjVQJdIrDJ0&t=2735s - Ryan Porter: The OpenAI Decisions API Needs a Confidence You Can Trust | Anthus. https://anth.us/blog/openai-decisions-api-preview/ - Amir Rafe amir rafe edu: Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with…. https://arxiv.org/html/2609.24052 - Kevin David Hayes: Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models. https://arxiv.org/html/2609.24881 - OpenAI et al.: GPT-4 Technical Report. https://arxiv.org/abs/2303.08774 - Guo et al.: On Calibration of Modern Neural Networks. https://arxiv.org/abs/1706.04599 - Kadavath et al.: Language Models (Mostly) Know What They Know. https://arxiv.org/abs/2207.05221 - Wikimedia Foundation, Inc.: Brier score - Wikipedia. https://en.wikipedia.org/wiki/Brier_score - TypeSafe AI docs: AI primer. https://docs.typesafe.ai/introduction/machine-learning-primer - The TWIML AI Podcast with Sam Charrington: Asking JEV's Creator the Hard Questions About This Hot New AI Model (at 55:53). https://www.youtube.com/watch?v=9Aato-NfjoU&t=3353s License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 10: Confidence and thresholds > The answer tells you what; the probability tells you whether to act. Here is the three-path pattern (act, confirm, escalate), why thresholds should differ by action, a one-line formula that turns the cost of mistakes into a threshold, and the one thing thresholds cannot protect you from. Course: System One (https://learn-system-one.pages.dev/) · Module 3: Probabilities you can act on · 4 min · Watch: https://learn-system-one.pages.dev/lessons/confidence-thresholds/ ## Key takeaways - Use probability as a second axis: the answer says what to do, the probability says whether to do it now. - Three paths: act automatically, ask to confirm, or escalate to a person or a slower model. - Thresholds belong to actions, not to the model: a read-only action can go at 0.6, a money transfer may need 0.85 and a confirmation. - With calibrated probabilities, act when p exceeds cost of a false action ÷ (cost of a false action + cost of a missed one). - Confidence can't flag inputs the model doesn't understand at all; check that inputs are in scope before you trust the number. ## Practices for coding agents - Gate every automated action on its own threshold, set by the cost of acting wrongly versus missing a true case. - Keep thresholds as named constants in code with a comment explaining their cost reasoning, and log the probability with each action. - Send the middle band to confirmation or human review instead of guessing. - Detect out-of-scope inputs (wrong language, unknown domain) before the decision, because confidence will not catch them. ## Lesson transcript ### Intro (0:01) A probability on its own doesn't do anything. Your code turns it into behavior with thresholds. This lesson is about where to put them, and why. ### A second axis (0:11) Here's the idea in one line, from the TypeSafe docs. Use confidence as a second axis. The answer tells you what. Confidence tells you whether to act. ### Three paths (0:23) That gives you three paths. When the model is sure, act automatically. When it's fairly sure, ask the user to confirm, or gather more information. And when it isn't, hand the case to a person, or to a slower, smarter model. Watch where each request goes. ### Thresholds per action (0:41) But one threshold for everything is a mistake. Thresholds belong to actions. In this voice banking example, checking a balance is harmless if it's wrong, so point six is enough. Approving a transfer is not, so it needs point eight five to go through, and anything less gets a spoken confirmation. And below point six, nothing happens without a person. ```python intent = answers["intent"] if intent.confidence < 0.6: route_to_support_agent(account) # genuinely unsure elif intent.choice == "check_balance": show_balance(account) # low stakes: 0.6 is fine elif intent.choice == "approve_transfer": if intent.confidence > 0.85: approve_transfer(account) # high stakes, high confidence else: ask_user_to_confirm(account) # high stakes, moderate ``` ### A threshold from costs (1:06) Where should a threshold sit? There's a formula, from decision theory. If acting on a false yes costs C false, and missing a true yes costs C miss, act when the probability is above C false divided by C false plus C miss. It assumes the probability is calibrated, which is why lesson nine came first. Two examples. Auto-approving a refund that wasn't owed costs you forty dollars, and making a real refund wait for review costs about ten. Forty over fifty: act above point eight. Now a safety report. A false alarm costs a minute of someone's time, a missed one could cost a hundred times more. One over a hundred and one: flag anything above about point oh one. Developer Daniel Sam put the attitude well. Fifty percent is a coin toss you ignore. Ninety five percent is a bet you take. Thresholds should scale with the cost of being wrong. ### Who owns the threshold (2:09) Some see this as a catch. Armin Ronacher put it as delegating the hallucination problem a little bit to the user. He's right, and that's the deal. The threshold is a business decision, and now it's yours: written in code, readable, and changeable in one line. ### What thresholds can't catch (2:27) One thing thresholds can't do. The author of Laya, an open decision model, fed its English model text in other languages. Accuracy collapsed, all the way to zero on some, but its average confidence never dropped below point eight eight five. A model can be confidently lost. Check that inputs are in scope before you trust the number. ### Quick check (2:50) **Quiz:** A wrong auto-close costs 9× a needless review. Threshold? A. 0.5 B. 0.75 C. 0.9 D. 0.99 Answer: C. C_false ÷ (C_false + C_miss) = 9 ÷ (9 + 1) = 0.9. Auto-close above 0.9; review the rest. ### Try it (3:07) Try it. Have your agent write the threshold policy for one decision, with the cost reasoning in the code. ```text For one automated decision in this codebase, write a threshold policy: - list each action it can trigger and estimate the cost of a wrong action vs a missed one (ask me if you can't infer them) - compute each threshold as C_false / (C_false + C_miss) - add a review band that goes to a person or a slower model - add a check that rejects out-of-scope inputs before the decision Put thresholds in named constants with the cost reasoning in comments, and log probability, threshold and action for every decision. ``` ### Recap (3:14) So: the answer says what, the probability says whether. Act, confirm, or escalate. Give every action its own threshold, set by what mistakes cost. And check that inputs are in scope, because a model can be confidently lost. Next: how much can you automate, and how to pick thresholds from data. ## Sources - TypeSafe AI docs: Confidence-gated routing. https://docs.typesafe.ai/patterns/confidence-routing - TypeSafe AI docs: Confidence. https://docs.typesafe.ai/confidence - elkan-cost-sensitive. https://cseweb.ucsd.edu/~elkan/rescale.pdf - Daniel Sam: The Model That Doesn't Talk. https://dev.to/danielsamfdo/the-model-that-doesnt-talk-5adm - Flavio Copes: How to build typed decisions with Jev. https://flaviocopes.com/jev/ - The TWIML AI Podcast with Sam Charrington: Asking JEV's Creator the Hard Questions About This Hot New AI Model (at 50:55). https://www.youtube.com/watch?v=9Aato-NfjoU&t=3055s - IBM Technology: What Is Jev? The AI Model That Doesn't Generate Text (at 12:16). https://www.youtube.com/watch?v=YGgNBcIgI4s&t=736s - InfoQ: TypeSafe AI Releases Jev: a Decision-Only Model That Returns Typed Probabilities. https://www.infoq.com/news/2026/10/typesafe-ai-jev-released/ - Laya — 33ms Multilingual System 1 Decision Engine. https://laya.convaiinnovations.com/ - IBM Technology: Building Decision Agents with LLMs & Machine Learning Models (at 10:35). https://www.youtube.com/watch?v=mRkJTXDromw&t=635s - Ryan Porter: The OpenAI Decisions API Needs a Confidence You Can Trust | Anthus. https://anth.us/blog/openai-decisions-api-preview/ License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 11: Automate the sure ones > You don't need a model that is right about everything. You need to know which answers you can trust. Here is the coverage curve that answers 'how much can we automate at 95%?', how to pick thresholds from a few hundred labels, what confidence-ordered review saves on our own data, and how to answer less precisely instead of wrongly. Course: System One (https://learn-system-one.pages.dev/) · Module 3: Probabilities you can act on · 4 min · Watch: https://learn-system-one.pages.dev/lessons/selective-automation/ ## Key takeaways - The real question is: what share of decisions can we automate at the accuracy we need? - A coverage curve answers it: on our banking data, Jev was 80% right overall, but 96% right on the half it was surest about. - Pick thresholds from 200 to 500 labeled examples by simulating each cutoff; treat the result as an estimate. - Reviewing the least confident answers first reached 95% system accuracy with 33% human review, versus 75% for random review. - When unsure, answer less precisely: falling back to a broader category turned 40% right into 70% right in TypeSafe's SEC example. ## Practices for coding agents - Build a coverage table (threshold, share automated, accuracy) from labeled examples before choosing a threshold. - Route the least confident answers to people first, and size the review queue from the coverage table. - When a taxonomy is hierarchical, return the parent category instead of a guess when confidence is low. - For rare positive classes, judge thresholds by precision and recall on the flagged items, not overall accuracy. ## Lesson transcript ### Intro (0:01) No model is right about everything. But you don't need that. You need to know which of its answers you can trust, and send the rest somewhere else. That's selective automation. ### The real question (0:13) On launch day, a Hacker News commenter asked the question that matters. Set a confidence threshold, trust the decisions above it, reject the rest. Then ask: what percent of decisions can we automate, and still hit ninety percent accuracy? ### The coverage curve (0:29) Here's that answer for Jev on our 3,080 banking messages. Automate everything, and it's eighty percent right. Only automate answers above point nine, and you handle sixty nine percent of messages at ninety two percent. Above point nine nine: half of all messages, at ninety six percent. To hit ninety five, you can automate about fifty seven percent. ### Picking a threshold (0:53) How do you find your curve? Shreya Shankar, who researches LLM pipelines, uses a simple recipe. Take a sample of real examples; she uses five hundred. Label them. Run the model and record its probabilities. Then simulate every cutoff, and pick the one that meets your accuracy target with the most automation. It's an estimate from a sample, so leave some margin. ### Review the doubtful first (1:18) Probabilities also tell people where to look. Suppose reviewers fix every answer they check. To get the whole system to ninety five percent, reviewing a random sample means checking three out of every four messages. Reviewing the least confident first, it's one in three. ### Ranking beats accuracy (1:36) Which means the property that matters most isn't raw accuracy. It's whether wrong answers come with low probability. One independent test on the same banking set found a fine tuned model whose accuracy moved by under a point, while the share it could handle on its own jumped from sixty four to ninety two in a hundred. ### Answer less precisely (1:57) There's another option besides asking a person: answer less precisely. TypeSafe classified sixty company filings into seventy five industry groups. Where the model was sure, it was right twenty seven times out of thirty. Where it wasn't, only twelve. But reporting those uncertain ones at the broader division level took them to seventy percent. Same request, no second call. ### Honest uncertainty (2:21) This is the deeper point of the whole module. The TypeSafe docs put it plainly: if an intelligent system, whether human or machine, cannot express honest uncertainty, the system cannot be trusted. I don't know is a feature. Build paths for it. ### Rare cases (2:39) One trap. When the thing you're looking for is rare, accuracy lies. If one percent of messages report fraud, a rule that never flags anything is ninety nine percent accurate, and useless. For rare classes, judge thresholds by precision and recall on the flagged items, and count how many a person must read. ### Quick check (3:02) **Quiz:** ≥ 0.95 covers 60% of traffic at 97%. Target is 95%. The other 40%? A. Automate them too; the average is still fine B. Route them to review, least confident first, or answer them less precisely C. Drop them D. Lower the threshold until everything is automated Answer: B. Automate the sure 60%. Send the rest to people (least confident first) or to a coarser answer, and re-measure as traffic changes. ### Try it (3:20) Try it. Have your agent build a coverage table for one of your decisions, and turn it into a threshold and a review budget. ```text Using 300 labeled examples of one decision, run the model and record the probability of its answer. For thresholds 0, 0.5, 0.7, 0.8, 0.9, 0.95, 0.99, print: share automated, accuracy on the automated share, and how many items per day would go to review at our current volume. Recommend the threshold that meets a 95% target with the most automation, and explain the margin you left. If there's a parent category, also report accuracy when unsure answers fall back to it. ``` ### Recap (3:28) So: ask how much you can automate at the accuracy you need, and read it off a coverage curve built from your own labels. Review the doubtful first. Answer less precisely when you can. And watch out for rare classes. That's module three. Next: composing many small judgments. ## Sources - Hacker News commenters: Hacker News: Introducing System One Models and Jev. https://news.ycombinator.com/item?id=49717558 - TypeSafe AI docs: Classification using confidence. https://docs.typesafe.ai/cookbooks/classification_using_confidence - TypeSafe AI docs: Confidence. https://docs.typesafe.ai/confidence - Hamel Husain: How to Cut Your LLM Classification Costs by 90% (at 9:40). https://www.youtube.com/watch?v=FWFjLF_VoVI&t=580s - Model Fatigue: We tested JevBench's top three against Jev on 3,080 bank messages (at 6:49). https://www.youtube.com/watch?v=nyl9IQbk5Zk&t=409s - jaredpalmer/kev: jaredpalmer/kev: Jev-like family of decision models built on top of Qwen3.5/3.8 you can train and run on…. https://github.com/jaredpalmer/kev - Geifman, El-Yaniv: Selective Classification for Deep Neural Networks. https://arxiv.org/abs/1705.08500 - Kevin David Hayes: Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models. https://arxiv.org/html/2609.24881 - Amir Rafe amir rafe edu: Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with…. https://arxiv.org/html/2609.24052 - Emil Lindfors: An early-access test of TypeSafe's Jev: calibrated judgments for half a cent. https://lindfors.no/blog/a-first-look-at-typesafes-jev/ License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 12: Decompose: many small questions > TypeSafe calls it the most important concept in its guide: break one broad judgment into narrow questions and combine the answers in code. Here is what decomposition buys you (reasons, control, per-signal thresholds), why decision science has recommended it for seventy years, and the case where it made things worse. Course: System One (https://learn-system-one.pages.dev/) · Module 4: Composing judgments · 4 min · Watch: https://learn-system-one.pages.dev/lessons/decompose-judgments/ ## Key takeaways - A broad question hides several judgments behind one number; narrow questions expose them so you can inspect, tune and combine them. - The broad answer is often right, but only the narrow ones tell you why: which signal fired, which tool argument was wrong. - Combine in code: weighted sums for factors that trade off, separate hard rules for 'any serious violation'. - Decision science agrees: simple formulas over separately judged criteria beat holistic judgment (Meehl, Dawes, Kahneman, the Apgar score). - Decomposition is not magic: it can't turn a System Two problem into System One answers, so validate the combination on labeled data. ## Practices for coding agents - Split broad judgments into narrow, independent questions, each a few-second call for an expert; ask them in one request. - Combine answers in code with explicit weights or rules; keep the raw per-question answers for debugging and explanations. - Use separate conditions for veto rules instead of letting a weighted sum average a serious violation away. - Check on labeled data that the decomposed version actually beats the single question before switching. ## Lesson transcript ### Intro (0:01) If you remember one technique from this course, make it this one. Don't ask one big question. Ask several small ones, and combine the answers in code. ### Small questions (0:12) Take spam. You could ask one question: is this message spam? Point nine seven. Fine, but why? Split it into narrow questions instead. Does it ask for a password? Point nine nine. Does it promise an unexpected reward? Point nine seven. Does it pressure you to act fast? Point nine eight. Does the sender's name clash with their domain? Point seven two. Now combine them in code, with weights you control. The TypeSafe guide is blunt about it. This is probably the most important concept in this guide. Broad questions hide several judgments behind one answer. Atomic questions expose those judgments, so you can inspect, tune, and combine them in code. ### Reasons, not just verdicts (0:59) Here's where it pays off. An agent was asked for tomorrow's weather in Seattle, in Fahrenheit. Ask the broad question, are these tool calls correct, and you get point one. Something's wrong, but what? Ask six narrow questions, and five come back near one. One doesn't: does the unit match the request? Point oh two. The agent asked for Celsius. Simon Willison worried, when Jev launched, that it's a black box. If it marks something as spam, which signals tipped it off? Decomposition is the answer. Each narrow question is a reason you can log, show to a reviewer, and test on its own. ### Combine in code (1:40) Then combine them in code. For factors that trade off against each other, use a weighted sum. But don't let a weighted sum average away a serious violation. If a message asks for credentials, that should block it on its own, whatever else is true. Rules and weights both live in your code. ```python a = {k: v.noul for k, v in response.answers.items()} if a["requests_credentials"] > 0.9: # a veto, not a weight return quarantine(msg, reason="asks for credentials") risk = (0.45 * a["offers_unexpected_reward"] + 0.30 * a["sender_identity_mismatch"] + 0.25 * a["creates_time_pressure"]) if risk > 0.6: return quarantine(msg, reason=top_signals(a)) ``` ### Seventy years of evidence (2:00) This isn't new. Since Paul Meehl's work in the 1950s, study after study has found that simple mechanical rules beat expert intuition at many prediction tasks. As decision researcher Olivier Sibony puts it: even the simplest rule will typically make fewer mistakes than the humans who think they are much smarter than the simple rule. The classic example is the Apgar score, from 1952. Instead of a doctor's overall impression of a newborn, five quick judgments: appearance, pulse, grimace, activity, breathing. Each zero to two, added up. Seven or more is normal. Three or below means act now. Narrow judgments, a sum, and bands for action. That's the pattern. ### Where it fails (2:48) But decomposition is not magic. Mike Ramos tested Jev on poker decisions, where the right move depends on game theory. Six narrow judgments instead of one question changed forty seven actions. Twelve got better. Twenty one got worse. Poker strategy is a System Two problem, and splitting it into fast judgments didn't change that. ### Quick check (3:12) **Quiz:** Policy: any threat of violence means removal. How do you encode it? A. As one weight in a weighted risk score B. As its own Noul with a hard rule in code that removes on its own C. Ask one broad question: does this break the policy? D. Average three runs of the broad question Answer: B. A veto shouldn't be averaged away by other factors. Ask it as its own narrow question and enforce it with a separate rule in code. ### Try it (3:27) Try it. Have your agent decompose one broad judgment in your code, and compare it with the original on labeled data. ```text Pick one broad judgment we ask a model to make (spam, risk, quality, policy violation). Split it into 4-8 narrow questions, each a few-second judgment for an expert, asked in ONE request. Combine them in code: hard rules for vetoes, weights for factors that trade off. Then run both versions on 100 labeled examples and report accuracy, plus 5 examples where the narrow answers explain a decision the broad one got wrong (or right). ``` ### Recap (3:35) So: ask many small questions and combine them in code. You get reasons, not just verdicts, and control over weights and vetoes. Decision science has backed this for seventy years. But it's not magic: validate it on your data. Next: asking all those questions at once. ## Sources - TypeSafe AI docs: How to build with TypeSafe. https://docs.typesafe.ai/concepts/how-to-build-with-system-one - TypeSafe AI docs: Composite scoring. https://docs.typesafe.ai/patterns/composite-scoring - TypeSafe AI: TypeSafe agent skill (SKILL.md). https://github.com/typesafe-ai/skills/blob/main/skills/typesafe-ai/SKILL.md - Simon Willison: Jev introduces a new shape of LLM—System One, aka Decision Models. https://simonwillison.net/2026/Sep/21/jev/ - CFA Institute: Olivier Sibony: How can you overcome noise and bias in decision making? (at 11:20, 9:44). https://www.youtube.com/watch?v=wiYaXOGE_e4&t=680s - Talks at Google: Thinking, Fast and Slow | Daniel Kahneman | Talks at Google (at 18:35). https://www.youtube.com/watch?v=CjVQJdIrDJ0&t=1115s - DailyData: Statistical Prediction Rules (SPRs) vs. Clinical experts (at 1:53). https://www.youtube.com/watch?v=7IgUcZBq1ec&t=113s - Authority control databases International GND FAST National United States Israel Other Yale LUX: Apgar score - Wikipedia. https://en.wikipedia.org/wiki/Apgar_score - Mike Ramos: Jev is the 🐠 at the poker table. https://backnotprop.com/blog/jev-poker/ - Sam Witteveen: Jev - The Ultimate Classification Model? (at 4:26). https://www.youtube.com/watch?v=X117w2Rark8&t=266s - Diogo Almeida: Introducing System One Models & Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 13: Ask everything at once > Questions about the same state belong in one request: they are evaluated in parallel and independently, so extra questions add a few tokens and almost no time. Here is what we measured with 13 questions, why you should ask speculative questions you might not need, and the few cases where a second request is genuinely warranted. Course: System One (https://learn-system-one.pages.dev/) · Module 4: Composing judgments · 3 min · Watch: https://learn-system-one.pages.dev/lessons/speculative-fan-out/ ## Key takeaways - Every question in a request sees the same state and is evaluated independently and in parallel. - In our test, 13 questions in one request took 0.27 seconds and 1,828 tokens; 13 separate requests took 3.3 seconds and 18,940 tokens, with the same answers. - Ask speculative questions: everything your code might need, including ones that only matter on some branches, and let code ignore the rest. - A second request is warranted only when an earlier answer changes the state or the options of the next question. - Coding agents tend to write one question per call; check their integrations for this. ## Practices for coding agents - Batch every question about the same state into one request, including speculative ones for later branches. - Split into a second request only when the next question's state or options depend on an earlier answer. - Review agent-written integrations for one-question-per-call loops and merge them. ## Lesson transcript ### Intro (0:01) Once you're asking many small questions, the next question is how to send them. The answer is: all at once. Let's see why, with numbers. ### Parallel and independent (0:11) In one request, every question sees the same state, and each is evaluated on its own, in parallel. One answer is never hidden context for another. So adding a question costs a few tokens for its text, and almost no time. ### What we measured (0:27) We tested it. One support ticket, the customer's record, and a seven hundred word policy, about eighteen hundred tokens of state. Thirteen questions: the category, a refund request, a duplicate charge, whether policy allows it, a login problem, a repeat contact, a churn threat, and more. All thirteen came back together. In one request, thirteen questions took a quarter of a second and eighteen hundred tokens. As thirteen separate requests, one after another, they took three point three seconds and almost nineteen thousand tokens, because the state was sent thirteen times. Firing the thirteen in parallel fixes the time, but not the bill. And the answers? The same, within point oh four. ### Speculative questions (1:16) So ask speculatively. Include every question your code might need, even ones that only matter on some branches. Bug severity only matters if it's a bug report. A refund question only matters for billing. Ask them all anyway, and let the code use what applies. ```python questions = { "category": Choice(...), # always needed "bug_severity": Score(...), # only if category == bug "has_repro_steps": Noul(...), # only if category == bug "refund_requested": Noul(...), # only if category == billing "frustration": Score(...), # always useful } a = client.system_one(state=ticket, questions=questions).answers if a["category"].choice == "bug" and a["bug_severity"].score > 1.5: escalate(ticket) # the speculative answers were ready ``` ### When a second request is right (1:34) When is a second request justified? Only when an earlier answer changes what you'd send next. TypeSafe's cookbooks have three real cases. Skill suggestion ranks a hundred eighty two skills, then fetches the full text of the top three and judges them again. Structure recovery first decides which line breaks split sentences, then classifies the paragraphs that didn't exist before. And hierarchical classification uses each level's answer to pick the next level's options. ### Watch your agent (2:04) One warning from the TypeSafe docs: coding agents fall into the one question per call habit more than people do. If an agent writes your integration, look for a loop that sends one question at a time, and merge it. ### Quick check (2:19) **Quiz:** You need the category, and if billing, whether a refund was asked for. Best approach? A. Ask the category, then send a second request for billing tickets B. Ask both in one request; ignore the refund answer for non-billing tickets C. Ask one combined question: is this a billing refund request? D. Ask an LLM to write both answers as JSON Answer: B. The refund question doesn't depend on the category answer; it can be asked of any ticket. Ask it speculatively in the same request and let code decide whether to use it. ### Try it (2:33) Try it. Have your agent find and merge your one question per call loops, and measure the difference. ```text Find every place we call a decision model (or an LLM classifier) more than once about the same input. Merge the questions into one request, adding speculative questions that later branches might need. Keep a second request only where the next question's state or options depend on an earlier answer, and add a comment explaining why. Measure latency and input tokens before and after on 20 real inputs. ``` ### Recap (2:39) So: questions about the same state go in one request. They're evaluated in parallel and independently, so extra questions are nearly free. Ask speculatively, and only make a second request when the next question depends on the first. Next: routing requests to the right handler. ## Sources - TypeSafe AI docs: Speculative fan-out. https://docs.typesafe.ai/patterns/fan-out - TypeSafe AI docs: Primitives (Questions). https://docs.typesafe.ai/primitives - TypeSafe AI docs: Parallel questions. https://docs.typesafe.ai/cookbooks/parallel_questions - TypeSafe AI docs: Skill suggestion. https://docs.typesafe.ai/cookbooks/skill_suggestion - TypeSafe AI docs: Structure recovery. https://docs.typesafe.ai/cookbooks/autoformat - TypeSafe AI docs: Hierarchical classification. https://docs.typesafe.ai/cookbooks/hierarchical_classification - Flavio Copes: How to build typed decisions with Jev. https://flaviocopes.com/jev/ - AWS Strands Labs: strands-decider: inference. https://github.com/strands-labs/strands-decider/blob/main/docs/inference.md - Marc Brooker: Strands Decider: Why Not an Encoder?. https://brooker.co.za/blog/2026/10/04/encoders.html - TypeSafe AI: TypeSafe agent skill (SKILL.md). https://github.com/typesafe-ai/skills/blob/main/skills/typesafe-ai/SKILL.md License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 14: Route and cascade > One fast decision in front of expensive things. Here is intent routing (code, a specialist model, or a person), the cascade (cheap model first, escalate only the uncertain), how to tune it so it matches a strong model at a fraction of the cost, and the prompt-cache gotcha that turned one developer's model router into a money loser. Course: System One (https://learn-system-one.pages.dev/) · Module 4: Composing judgments · 4 min · Watch: https://learn-system-one.pages.dev/lessons/routing-and-cascades/ ## Key takeaways - Routing: one cheap, fast decision chooses the handler, which may be plain code, a specialist LLM, or a person. - If you can map the whole decision tree, build it explicitly and optimize each node; don't hand it to an agent loop. - A cascade answers with the cheap model when it's confident and escalates the rest to a stronger model or a person. - Tune the cascade on a sample of a few hundred items: pick the cheapest threshold that still agrees with the strong model at your target rate. - Route per task, not mid-session: switching models mid-conversation throws away the prompt cache. ## Practices for coding agents - Put one fast decision (intent + complexity) in front of expensive handlers, and send low-confidence routes to a person. - Implement cascades: accept confident cheap answers, escalate the rest, and tune the threshold on labeled data against the strong model's answers. - Prefer routing whole tasks or reasoning effort over swapping models mid-session, which invalidates prompt caches. ## Lesson transcript ### Intro (0:01) Expensive models are great, and wasteful when they're used for everything. The fix is old and simple: put one fast decision in front of the expensive things. ### Intent routing (0:13) Here's intent routing. A customer message comes in. One Choice decides the intent, and one Score estimates how complex it is, in a single fast call. Order status goes to plain code: a database lookup, no model at all. Product questions go to an LLM with product knowledge. Complaints that are complex, or unclear, go to a person. Anthropic's Barry Zhang gives the same advice for agents. If you can map out the entire decision tree pretty easily, just build that explicitly, and then optimize every node. It's cheaper, and it gives you more control. A decision model is how you make each node smart. ### The cascade (0:54) The cascade is routing by confidence. Every item goes to the cheap, fast model first. If it's confident, you keep its answer. If not, the item goes to a stronger, slower model, and only the leftovers reach a person. Most items never touch the expensive part. The savings can be dramatic. The FrugalGPT paper reported matching GPT-4's performance with up to ninety eight percent lower cost on its benchmarks. And the open model Jeff, on one Apple M4 Max, beat a twenty seven billion parameter model deciding directly, ninety five versus eighty seven percent, in under half a second instead of eight. ### Tuning it (1:36) How do you set the cascade's threshold? Shreya Shankar's recipe: treat the strong model as the oracle. Run both on a sample, say five hundred items. Simulate each threshold, measuring how often the cascade agrees with the oracle, and what it costs. Throw out every threshold below your target, and pick the cheapest one left. And if confidence carries no signal, the only threshold left sends everything to the oracle. The procedure tells you there's nothing to save, instead of pretending. ### Routing effort (2:07) You can route effort, not just models. The Jeff team put a small decision model inside a coding agent, deciding at each step whether the big model should think hard. Over twelve hundred tasks, it kept the same pass rate, sixty two percent, in about two thirds of the time. Simply turning thinking off was faster too, but seven and a half points worse. ### The cache gotcha (2:31) One gotcha from Hacker News. A developer used a decision model to route each turn of a session between small and large models, and lost money. Swapping models mid-session invalidates the prompt cache, which is where a lot of the savings live. Route whole tasks, or route the effort level, instead. ### Quick check (2:51) **Quiz:** Answers above 0.9 agree with the oracle no more often than answers below 0.5. What now? A. Raise the threshold to 0.99 B. Confidence carries no signal on this task: don't cascade it this way C. Lower the threshold to 0.5 D. Add more questions to the request Answer: B. A cascade only works if confidence tracks correctness. Here it doesn't, so any threshold routes at random. Improve the questions or the model, or send everything to the strong model. ### Try it (3:08) Try it. Have your agent put a router or a cascade in front of your most expensive model call. ```text Find our most expensive or slowest LLM call. Put a decision model in front of it: either a router (one Choice for intent + one Score for complexity, sending easy cases to code or a cheap path) or a cascade (accept confident answers, escalate the rest). Tune the threshold on 300 logged inputs using the expensive model's outputs as the oracle, and report agreement, cost and latency at each threshold. Route whole tasks, never switch models mid-session. ``` ### Recap (3:15) So: one fast decision in front of expensive things. Route to code, a specialist, or a person. Cascade: cheap first, escalate the uncertain, tuned against a strong model. Route effort, not just models, and never mid-session. Next: getting exact values out, without generating them. ## Sources - TypeSafe AI docs: Intent routing. https://docs.typesafe.ai/patterns/intent-routing - TypeSafe AI docs: SDE cascade. https://docs.typesafe.ai/cookbooks/sde_cascade - AnthropicAI: Building Effective AI Agents. https://www.anthropic.com/engineering/building-effective-agents - AI Engineer: How We Build Effective Agents: Barry Zhang, Anthropic (at 2:51). https://www.youtube.com/watch?v=D7_ipDqhtwk&t=171s - Hamel Husain: How to Cut Your LLM Classification Costs by 90% (at 4:43). https://www.youtube.com/watch?v=FWFjLF_VoVI&t=283s - Chen, Zaharia, Zou: FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. https://arxiv.org/abs/2305.05176 - Ong et al.: RouteLLM: Learning to Route LLMs with Preference Data. https://arxiv.org/abs/2406.18665 - firelex/jeff: firelex/jeff: Millisecond decisions, any domain: a 0.8B open "System 1" model that picks between your…. https://github.com/firelex/jeff - Hacker News commenters: Hacker News: Jev in 25 Lines of Python. https://news.ycombinator.com/item?id=49812769 - Hacker News commenters: Hacker News: Decisions API is in public beta. https://news.ycombinator.com/item?id=49984025 - Syntax: Jev Explained: Demos and Use Cases (at 4:44). https://www.youtube.com/watch?v=QbYBRjOaGOo&t=284s - Sebastien Dubois: Jev and System One models - a semantic if statement for your code. https://www.dsebastien.net/jev-and-system-one-models-a-semantic-if-statement-for-your-code/ License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 15: Select, don't generate > A decision model can't write a value, but most extraction is really selection: the value is already in the text. Let code find the candidates and let the model pick one, then copy it verbatim. Here is that pattern for phone numbers, dates, search reranking, line-level search and deep taxonomies. Course: System One (https://learn-system-one.pages.dev/) · Module 4: Composing judgments · 3 min · Watch: https://learn-system-one.pages.dev/lessons/select-dont-generate/ ## Key takeaways - Most extraction is selection: the value already exists in the source, so the model only has to pick it. - Code finds candidates (regex, parser, search), the model chooses, code copies the value verbatim, so nothing is invented. - Dates are parts: month, day and year are each a small Choice with a 'not stated' option; code assembles and does the arithmetic. - Reranking is selection too: a cheap search finds candidates, one Noul per candidate scores relevance. - Big taxonomies go level by level, keeping the best few paths instead of one greedy guess. ## Practices for coding agents - For extraction, generate candidates in code and ask a Choice which candidate is the requested value; always include a none option. - Extract dates as separate Choices for each part with 'not stated'; build and compare dates in code. - Rerank search results with one relevance question per candidate and sort by probability. - Walk hierarchical taxonomies level by level with a small beam of top paths. ## Lesson transcript ### Intro (0:01) A decision model can't write. So how do you get a phone number, a date, or the right paragraph out of a document? You don't ask it to write the value. You let it choose. ### The pattern (0:13) Here's the pattern. Most extraction is really selection, because the value is already sitting in the text. So code finds the candidates, with a regular expression, a parser or a search. The model picks the one you asked for. And code copies it verbatim. Nothing can be invented, because the model only ever chooses. ### Pick a value (0:35) This email signature has three phone numbers: a finance desk, a direct line, and a support hotline. A regex finds all three. Then one Choice: which candidate is the twenty four seven support number? The hotline, at one point oh. Your code copies that exact string, and there's no way to get a digit wrong. ### Dates are parts (0:56) Dates work the same way, part by part. Jev reads dates as text, not as numbers, so don't ask it which date is earlier. Ask three small Choices: which month, which day, which year, each with a not stated option. Here, March, the third, twenty twenty six. Then code builds the real date, and does every comparison. The model's job ends at the parts. Building the date, checking it's valid, computing a renewal or a deadline, comparing two dates: that's arithmetic, and arithmetic belongs in code. If any part comes back not stated, or with low confidence, ask a person. ```python m, d, y = (a[k].choice for k in ("start_month", "start_day", "start_year")) if "not_stated" in (m, d, y) or min(a[k].confidence for k in a) < 0.8: return send_to_review(doc) start = date(int(y), MONTHS.index(m) + 1, int(d)) # validates, too renewal = start.replace(year=start.year + 1) days_left = (renewal - today).days # code, not the model ``` ### Rerank (1:38) Search is selection too. In a TypeSafe cookbook, a cheap keyword search found thirty candidate passages for each of forty legal questions. The right one was always in the list, but it came first only five percent of the time. One Noul per candidate, is this passage relevant, then sort by probability. Now the right passage came first eighteen percent of the time, and in the top ten sixty two percent. ### Find the line (2:05) You can even point at lines. Another cookbook numbers the two hundred eighteen lines of GitHub's terms of service, and uses them as the options of a single Choice: which line answers this question? Plus one Noul: does the document answer it at all? Search without generating a word. ### Deep taxonomies (2:24) And for taxonomies with thousands of leaves, like patent classes or product categories, go level by level. At each level, a Choice picks among the children. Keep the best few paths, not just one, so a near miss early on can recover. That's a beam search over the model's probabilities. ### Quick check (2:43) **Quiz:** Get the invoice total from extracted PDF text. The select-don't-generate way? A. Ask a Score for the amount, from 0 to 10,000 B. Regex every money amount, ask a Choice which one is the total (plus “none”), copy it C. Ask a Noul for each possible number from 1 to 10,000 D. Ask the model to type the total Answer: B. Code finds the candidates, the model picks the one that's the total, and code copies it exactly. The model never has to produce digits. ### Try it (2:56) Try it. Have your agent replace one generated extraction in your code with find, choose, copy. ```text Find one place where we ask an LLM to extract a value (an amount, email, ID, date, name) from text. Replace it with select-don't-generate: candidates from regex/parsing in code, one Choice that picks the requested candidate (always with a "none" option), and code that copies the chosen string verbatim and normalizes it. For dates, ask month/day/ year as separate Choices with "not_stated". Compare both versions on 50 real documents. ``` ### Recap (3:04) So: when you need a value, select it, don't generate it. Code finds candidates, the model picks, code copies. Dates are parts. Reranking and line search are selection. And deep taxonomies go level by level. Next: using decision models to check other models' work. ## Sources - TypeSafe AI docs: Pre-parsed value extraction. https://docs.typesafe.ai/cookbooks/pre_parsed_value_extraction_cookbook - TypeSafe AI docs: Date extraction. https://docs.typesafe.ai/cookbooks/date_extraction_cookbook - TypeSafe AI docs: Re-ranking. https://docs.typesafe.ai/cookbooks/rerank_typesafe - TypeSafe AI docs: Line-by-line search. https://docs.typesafe.ai/cookbooks/semantic_find - TypeSafe AI docs: Hierarchical classification. https://docs.typesafe.ai/cookbooks/hierarchical_classification - TypeSafe AI docs: Structure recovery. https://docs.typesafe.ai/cookbooks/autoformat - TypeSafe AI docs: Jev 1.13 jaggedness. https://docs.typesafe.ai/model-jaggedness/jev-1.13 - Simon Willison: Jev introduces a new shape of LLM—System One, aka Decision Models. https://simonwillison.net/2026/Sep/21/jev/ - TypeSafe AI: TypeSafe agent skill (SKILL.md). https://github.com/typesafe-ai/skills/blob/main/skills/typesafe-ai/SKILL.md License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 16: Verify and guard > Decision models are cheap and fast enough to check everything: every citation an LLM writes, every message going in and out, every extracted field. Here is a citation check, a one-request guardrail screen, why two independent checks beat one, and why none of this is a security boundary. Course: System One (https://learn-system-one.pages.dev/) · Module 4: Composing judgments · 4 min · Watch: https://learn-system-one.pages.dev/lessons/verification-and-guardrails/ ## Key takeaways - Use decision models as fast judges: check claims against sources, screen messages, sanity-check extracted values. - One request can screen a message for several hazards plus a severity score, mapped by code to pass, review, block or route. - Your policy lives in your code, consistent across model versions, instead of in each lab's moving refusal line. - When code can check something exactly (a verbatim quote, a sum), let code do it. - A classifier guard reduces risk but is not a security boundary; with private data, untrusted input and outbound access, remove a capability. ## Practices for coding agents - Check LLM citations with a Choice (supports / partially / contradicts / unrelated) against the source context, and verify verbatim quotes with string matching in code. - Screen inputs and outputs with one request of narrow hazard questions plus a severity Score; map them to pass, review, block or route in code. - Combine independent checks (extraction confidence and plausibility) and keep both reasons when both fail. - Never rely on a decision-model guard alone to stop prompt injection; limit the agent's capabilities instead. ## Lesson transcript ### Intro (0:01) Language models write. Decision models can check what they wrote, fast and cheap enough to check every single output. This lesson is about judges and guards. ### Check the citation (0:12) Say an LLM writes: walking seven thousand steps a day cuts hospitalization risk by thirty one percent, the study proved. And it cites a quote. One Choice checks the claim against the quote in the context of the source. The answer: partially, at point nine nine. The number is right, but the source is an observational study that proved nothing. But notice what we didn't ask the model. Is the quote in the source word for word? We tried, and got point eight six, for a quote that is there exactly. That's a string match, and code gets it right every time. Let the model judge meaning. Let code check text. ```python def check(claim, quote, source): if quote not in source: # exact: code's job return "misquoted" a = decide(state={"claim": claim, "quote": quote, "source": source}, questions={"support": SUPPORT_CHOICE}).answers return a["support"].choice # meaning: the model's job ``` ### Guard the gate (0:54) Guardrails work the same way. Screen every message going into your LLM app with one request: a few narrow hazard questions, and a severity score. This message tries a prompt injection and pastes a card number. Injection, point nine nine. Card number, point nine eight. Self harm and harassment, near zero. Severity: must be blocked. Then code maps the answers to an action. Clean messages pass. Uncertain ones go to review. Clear violations are blocked. And some get routed: a card number gets redacted before the LLM ever sees it. ### Your policy, in your code (1:33) Why not just rely on the LLM's own refusals? The TypeSafe guardrails cookbook explains: labs teach most LLMs to refuse a set of unsafe requests, but each lab draws that line somewhere else, and each new version of a model moves it again. A separate judge puts your policy in your code. ### Two checks, two reasons (1:53) For extracted data, use two independent checks. Unsiloed reads a medication list and scores each field, then asks Jev whether each dose is plausible. A coffee stain over one dose fails the reading check, while Jev, seeing only the text, calls it plausible. Ten ibuprofen tablets three times a day reads perfectly, but Jev puts its plausibility at point oh four. Each check flags a review need the other misses. ### Not a security boundary (2:23) Now the warning. TypeSafe's notes say the state is data, but Jev doesn't treat it as hostile by default, and text written to steer it can move the answer. Our injection test was caught, but don't build on that. If an agent can read private data, see untrusted content, and send data out, Simon Willison's lethal trifecta, no classifier makes it safe. Remove one of the three. ### Quick check (2:48) **Quiz:** Does the LLM's quote appear in the source word for word? Best check? A. A Noul: “Is the quote in the source verbatim?” B. A Score for similarity C. A string match in code D. Ask the LLM to double-check itself Answer: C. Exact text checks are code's job. Use the decision model for meaning: does the quote, in context, support the claim? ### Try it (3:02) Try it. Have your agent add one judge and one guard to your LLM feature. ```text For our main LLM feature, add: 1. An input guard: one decision-model request with narrow hazard Nouls (injection, PII, abuse, out of scope) + a severity Score, mapped in code to pass / review / block / route (e.g. redact PII). 2. An output judge: for each claim with a citation, check the quote exists verbatim in code, then ask a Choice whether the quote, in context, supports / partially supports / contradicts the claim. Log every verdict with its probabilities, and test on 30 examples. ``` ### Recap (3:08) So: decision models make cheap, fast judges. Check meaning with the model and text with code. Screen messages with one request and map the answers to actions. Use independent checks. And remember, a guard is not a security boundary. That's module four. Next: decision models inside agents. ## Sources - TypeSafe AI docs: Double-checking citations. https://docs.typesafe.ai/cookbooks/citation_check - TypeSafe AI docs: Guardrails for LLMs. https://docs.typesafe.ai/cookbooks/llm_guardrails - TypeSafe AI docs: Jev 1.13 jaggedness. https://docs.typesafe.ai/model-jaggedness/jev-1.13 - Unsiloed AI: Document Review With Unsiloed and TypeSafe Jev. https://www.unsiloed.ai/blog/document-review-unsiloed-typesafe-jev - Simon Willison: The lethal trifecta for AI agents: private data, untrusted content, and external communication. https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/ - Diogo Almeida: Introducing System One Models & Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev - IndyDevDan: 10 Levels of Jev For Agentic Engineers (at 2:20). https://www.youtube.com/watch?v=_U-O5lYhJ7Q&t=140s - AI Engineer: $1 AI Guardrails: The Unreasonable Effectiveness of Finetuned ModernBERTs – Diego Carpintero. https://www.youtube.com/watch?v=YZHPEkfy2kc - Hamel Husain: Using LLM-as-a-Judge For Evaluation: A Complete Guide – Hamel’s Blog. https://hamel.dev/blog/posts/llm-judge/ License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 17: Decision models inside agents > Agents make dozens of small decisions per task: which tool, which skill, are these arguments grounded, is it time to ask the user, is the task done. Here is where a fast decision model fits in an agent loop, a gate that stops an agent from guessing, what skill selection gained in TypeSafe's test, and real-time control at ten decisions a second. Course: System One (https://learn-system-one.pages.dev/) · Module 5: Real systems · 4 min · Watch: https://learn-system-one.pages.dev/lessons/decision-models-in-agents/ ## Key takeaways - Most of an agent's decisions are rote; let a decision model make those and save the LLM for the hard ones. - Good spots: model routing, tool and skill selection, argument checks before a tool runs, guardrails, and 'is it done?' checks. - A before-tool-call gate with two narrow questions can make an agent ask instead of guessing. - Selecting skills with a decision model cut wrong skill loads from 16.8% to 7.3% in TypeSafe's test of 182 skills. - Prefer workflows (code-defined paths with smart nodes) when you can map the decision tree; use agents when you can't. ## Practices for coding agents - Gate tool calls on narrow questions (are the arguments grounded in what the user said? is it premature to act?) and guide the agent back instead of hard-failing. - Use a decision model to pick the few relevant tools or skills to load per step, instead of loading everything into context. - Give long-running agents a cheap 'ask the decision model' tool for yes/no and classification checks over files or pages. - Set a deadline for real-time decisions with a deterministic fallback when the answer is late. ## Lesson transcript ### Intro (0:01) Agents are where language models make the most decisions per minute: which tool, which file, ask or act, done or not. Most of those decisions are small. That's where a fast decision model belongs. ### Plus, not versus (0:15) TypeSafe's founder is clear that this isn't a rivalry. The future, he says, is not going to be System One models versus LLMs. It's System One models plus code. And for agents, plus the LLM. ### Where it fits (0:30) AWS built Strands Decider for exactly this. Its README lists the spots: route each task to the right model. Pick the next tool. Check a tool call's arguments before it runs. Triage, guardrails, cheap evaluations. And hybrid agents, where the LLM makes the hard decisions and a decider makes the rote ones. ### A gate before tools (0:53) Here's the Strands example. The agent has a weather tool, and an eager system prompt. A user asks: what's the weather? No city. The agent guesses one and calls the tool. But before the call runs, a hook asks the decider two narrow questions. Are the argument values grounded in facts the user actually gave? Is it premature to call this tool before clarifying? A few lines of Python, and the agent goes back and asks which city. ```python class GroundingGate(InterventionHandler): def before_tool_call(self, call, conversation): a = decider.ask( state={"conversation": conversation, "call": call}, questions={ "grounded": Noul("Are the argument values grounded " "in facts the user provided?"), "premature": Noul("Is it premature to call this tool " "before clarifying with the user?"), }) if a["grounded"] < 0.5 or a["premature"] > 0.5: return Guide("Ask the user which city they mean.") return Proceed() ``` ### Pick the right skill (1:22) Skill selection is another big one. Agents usually stuff every skill description into the prompt and hope. TypeSafe tested an agent with a hundred eighty two skills. On its own, it loaded the wrong skill for sixteen point eight percent of requests that had a matching skill. With a decider ranking skills first, then rechecking the top three with their full text, that fell to seven point three percent. Needless loads fell from nine point eight to four. ### Ask the decider (1:51) You can also hand the agent the decision model as a tool. Developer IndyDevDan gave his coding agent an ask Jev tool for yes or no and classification questions about files. The agent stopped reading whole files into its expensive context just to answer a simple question. His words: we delegated a QA task for file reading outside of my expensive language model to Jev. ### Real time (2:17) And speed opens a new kind of agent: one that acts in real time. TypeSafe's demo plays Doom at ten decisions a second, which costs about seven dollars an hour. Opper's Pac-Man benchmark gives each decision a two second deadline: answer late, and a simple backup rule moves instead. ### Workflow or agent? (2:36) Finally, step back. Anthropic separates workflows, where code defines the paths, from agents, where the LLM decides its own steps. A decision model makes workflows far more capable, because every node can now judge meaning. So before you build an agent, ask: could this be a workflow with smart nodes? If you can map the decision tree, it should be. ### Quick check (3:01) **Quiz:** An agent calls a payment tool with an amount the user never said. Best fix? A. Tell the agent in the prompt to be careful B. A before-tool-call gate: “Is the amount grounded in what the user said?” → guide it back to ask C. Ask the decider to choose the amount D. Lower the model temperature Answer: B. A narrow grounding question before the call catches invented arguments, and guiding the agent back makes it ask the user instead of guessing. ### Try it (3:15) Try it. Have your agent add a decision model gate to its own tool calls. ```text In our agent, add a hook that runs before every tool call. It sends the conversation and the proposed call to a decision model with narrow questions: are the argument values grounded in what the user said? is it premature to act before clarifying? is this tool appropriate for the request? If any check fails, hand the turn back to the agent with specific feedback instead of failing. Log the probabilities, and add a test where the user omits a required detail. ``` ### Recap (3:20) So: decision models go inside agents, not instead of them. Route, pick tools and skills, check arguments, decide when it's done. Gate risky calls and guide the agent back. And when you can map the decision tree, build a workflow with smart nodes. That's module five's first half. Next: turning judgments into data. ## Sources - Modal: jev: on | Diogo Almeida founder of Typesafe AI, Runtime 2026 (at 6:53). https://www.youtube.com/watch?v=ip7Qej0ONoE&t=413s - AnthropicAI: Building Effective AI Agents. https://www.anthropic.com/engineering/building-effective-agents - AWS Strands Labs: strands-decider (README). https://github.com/strands-labs/strands-decider - AWS Strands Labs: strands-decider: agent example. https://github.com/strands-labs/strands-decider/blob/main/examples/strands_README.md - Marc Brooker; Mike Chambers; Fabio Nonato de Paula: Introducing Strands Decider 2B: a small, open source, decision model. https://strandsagents.com/blog/introducing-strands-decider/ - TypeSafe AI docs: Skill suggestion. https://docs.typesafe.ai/cookbooks/skill_suggestion - IndyDevDan: 10 Levels of Jev For Agentic Engineers (at 21:20). https://www.youtube.com/watch?v=_U-O5lYhJ7Q&t=1280s - Steve (Builder.io): Fake Jev demos are taking over the internet (at 7:36). https://www.youtube.com/watch?v=Spn-F83ZHH0&t=456s - Diogo Almeida: Introducing System One Models & Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev - Opper AI: JevMan: a Pac-Man benchmark for decision models | Opper AI. https://opper.ai/jevman-benchmark/ - Hacker News commenters: Hacker News: Strands Decider 2B: a small, open-source, decision model. https://news.ycombinator.com/item?id=49987076 - The TWIML AI Podcast with Sam Charrington: Asking JEV's Creator the Hard Questions About This Hot New AI Model (at 73:49). https://www.youtube.com/watch?v=9Aato-NfjoU&t=4429s License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 18: Judgments as data > At a fraction of a cent per answer, judgments stop being calls you make and become data you keep: columns you can sort, filter, weight and train on. Here is a study that coded half a million police narratives for $25, a loop that turned tasting notes into features for a classic model, and why cheaper judgment will mean much more of it. Course: System One (https://learn-system-one.pages.dev/) · Module 5: Real systems · 4 min · Watch: https://learn-system-one.pages.dev/lessons/judgments-as-data/ ## Key takeaways - Ask once, store the answers, and reuse them: changing a weight or a filter doesn't need another model call. - Whole corpora are now affordable: a study screened 499,500 police narratives for $25 and coded 195,857 with 27 questions for $30. - Judgments make great features: questions turned wine notes into 67 numeric columns, and a classic model beat every baseline. - With labels, let the data choose the questions: an autoresearch loop proposes questions and keeps the ones that cut error. - Cheaper judgment means more judgment, which is why TypeSafe named Jev after Jevons. ## Practices for coding agents - Store raw per-question answers (probabilities, model ID, date) as data, and recompute composites in code when weights change. - Before coding a large corpus, run a lean screening schema first and the full schema only where it applies. - When you have labels, use decision-model answers as features for a classic model, and keep only questions that improve held-out error. - Estimate cost per 1,000 items from measured token counts before launching a big run. ## Lesson transcript ### Intro (0:01) When a judgment costs a fraction of a cent, you stop thinking of it as a call. It becomes data: a column you compute once, store, and use in a hundred ways. ### Ask once, reuse (0:12) The habit is simple. Store the raw answers: every probability, with the model version and the date. Then your weights, thresholds and filters are just code over a table. Change a weight and the ranking updates instantly, with no new model calls. One tester put it neatly: the judgments are cached, so rewriting the composition rules costs nothing. ```text -- one row per item, one column per question (raw probabilities) CREATE TABLE ticket_judgments ( ticket_id TEXT, model TEXT, judged_at DATE, refund_requested REAL, duplicate_charge REAL, churn_threat REAL, frustration REAL); -- change the weights any time: no new model calls SELECT ticket_id, 0.5*churn_threat + 0.3*frustration/2 + 0.2*duplicate_charge AS priority FROM ticket_judgments ORDER BY priority DESC; ``` ### Whole corpora (0:36) That changes what's affordable. Texas has five million police crash narratives that analysts rarely read, because coding them with a frontier model is too expensive. A research team screened half a million of them with ten questions, for twenty five dollars. Then they coded almost two hundred thousand with twenty seven questions, for thirty dollars, with zero failed calls. The whole five million would cost about seven hundred seventy four. Notice the two stages. A lean screen with ten presence questions runs over everything. The detailed twenty seven question schema only runs where the screen found something to code. The raw probabilities overstated how common things were, but once recalibrated on human labels, the review budget became a number: how many records a person would still need to read. ### Judgments as features (1:27) Judgments also make great features for classic machine learning. A TypeSafe cookbook predicts a wine critic's score from the tasting note. Predicting the average gives an error of three point oh nine points. Word counts get two point four seven. Asking the decision model for the score directly, two point one five. But thirty eight narrow questions, fed as features to a CatBoost model, got it down to one point seven seven. ### Let the data pick the questions (1:54) And nobody wrote those thirty eight questions by hand. An LLM proposed them, the decision model answered them for every row, and CatBoost reported which questions helped and which rows it still got wrong. The next round read that report and proposed better questions. Five rounds took the error from one point eight seven to one point seven seven. Trying a change costs no API calls, because the answers are already stored. ### Jevons (2:21) Which brings us to the name. TypeSafe named Jev after William Stanley Jevons, who noticed in 1865 that more efficient steam engines made Britain burn more coal, not less. Their bet: every order of magnitude drop in the cost of intelligence unlocks orders of magnitude more uses. Whether that holds depends on demand, but it's the right way to think about judgments as data. ### Quick check (2:47) **Quiz:** 10,000 tickets were scored with 6 questions last week. New priority weights today. What now? A. Rescore all 10,000 tickets B. Recompute the priority in code from the stored answers C. Ask one new question per ticket D. Retrain the decision model Answer: B. The judgments didn't change, only how you combine them. If you stored the raw answers, a new weighting is a query, not a model call. ### Try it (3:02) Try it. Have your agent turn one judgment in your product into a stored, reusable table. ```text Pick one judgment we make about records (tickets, leads, reviews, documents). Create a table that stores the raw answer for each question (probabilities), the model ID and the date, keyed by record. Backfill 1,000 records in batches and report the measured cost per 1,000. Move the weighting and thresholds into a query or function over that table, and show that changing a weight needs no new model calls. ``` ### Recap (3:08) So: judgments are data. Store the raw answers with the model and date, and compute composites in code. Whole corpora are affordable now, so screen first, then code. Use judgments as features, and let the data pick the questions. That's module five. Next, module six: shipping it, starting with the jagged edges. ## Sources - Amir Rafe amir rafe edu: Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with…. https://arxiv.org/html/2609.24052 - TypeSafe AI docs: Autoresearch feature discovery. https://docs.typesafe.ai/cookbooks/autoresearch_feature_discovery - TypeSafe AI docs: Composite scoring. https://docs.typesafe.ai/patterns/composite-scoring - TypeSafe AI: TypeSafe agent skill (SKILL.md). https://github.com/typesafe-ai/skills/blob/main/skills/typesafe-ai/SKILL.md - Emil Lindfors: An early-access test of TypeSafe's Jev: calibrated judgments for half a cent. https://lindfors.no/blog/a-first-look-at-typesafes-jev/ - Diogo Almeida: Introducing System One Models & Jev. https://typesafe.ai/blog/introducing-system-one-models-and-jev - Wikimedia Foundation, Inc.: Jevons paradox - Wikipedia. https://en.wikipedia.org/wiki/Jevons_paradox - Mike Ramos: Jev is the 🐠 at the poker table. https://backnotprop.com/blog/jev-poker/ - Hacker News commenters: Hacker News: Introducing System One Models and Jev. https://news.ycombinator.com/item?id=49717558 License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 19: Know the jagged edges > Decision models are strong at semantic judgment and unreliable at things a calculator does. Here are the failure modes we reproduced (counting, arithmetic, dates), TypeSafe's own list of nine, the cases where the model is wrong without sounding unsure, and the rule that fixes most of them: reason in code, decide with the model. Course: System One (https://learn-system-one.pages.dev/) · Module 6: Shipping it · 4 min · Watch: https://learn-system-one.pages.dev/lessons/jagged-edges/ ## Key takeaways - Decision models are jagged: excellent at meaning, unreliable at counting, arithmetic and date comparison. - In our tests, one counting question got 5 fruits instead of 6; twelve per-item Nouls summed in code got it right. - On a wrong invoice total and a late payment, the model hedged at 0.43 and 0.47: honest, but code answers exactly. - Typed output can't be malformed, but it can still be wrong, sometimes with confidence. - Don't ask for counts, sums, date math, forecasts of real-world outcomes, or long reasoning chains; reason in code and decide with the model. ## Practices for coding agents - Never ask a decision model to count, add, compare numbers or compare dates; extract the parts and compute in code. - For semantic counting, ask one Noul per item and sum in code. - Write the exact condition in the instructions; when you find yourself explaining what you meant, add that to the question. - Test each new question on known hard cases (negations, look-alike options, adversarial text) before shipping. ## Lesson transcript ### Intro (0:01) Every model has edges where it suddenly gets worse. For decision models, those edges are sharp, and easy to predict. Let's walk right up to them. ### Jagged (0:11) TypeSafe's CEO doesn't hide it. Our model cannot count R's in strawberry, he says. Luckily, that is not what real world work looks like. Most work is semantic judgment. But you need to know exactly where the edges are. ### Counting (0:27) We checked. Twelve words, six of them fruits. Asked as one question, how many are fruits, the model said five, at fifty two percent. Wrong. Now ask twelve small Nouls, one per word: is this a fruit? Every one came back right, and code adds them up. Six. ### Arithmetic (0:47) Arithmetic. An insurance claim lists four costs adding up to three thousand two hundred fifty, but claims three thousand one fifty. Do the costs add up? The model said point four three. That's honest, it really doesn't know, but it's useless. One line of code knows for sure. ### Dates (1:08) Dates. An invoice from February tenth, due in thirty days, was paid on March fourteenth. Was it on time? Point four seven, a shrug. TypeSafe's docs explain why: the model reads dates as text, not as quantities. Extract the parts, as in lesson fifteen, and let code count the days. ### The full list (1:29) TypeSafe publishes the full list for Jev one point thirteen, nine failure modes, each with a fix. Literal reading. Math. Date comparison. Indirection. Large, noisy state. Adversarial content. Instructions that contradict the criteria. Option order. And generation, which it simply doesn't do. Several of these you've already met in this course. ### Wrong, and not unsure (1:55) Now the uncomfortable part. Low confidence won't always warn you. A poker player tested Jev on a spot every regular knows: you hold the best possible hand, so you check and let the other player bet. The solver checks every time. Jev went all in, sixteen runs out of sixteen. That's why the claim that a decision model can't hallucinate needs care. It can't return an invalid type or an option that doesn't exist. But as TypeSafe's CEO wrote on launch day, it's also possible to be confidently wrong, and all future models will still have that possibility. ### What not to ask (2:34) So here's what not to ask a decision model. Don't ask it to count, add, or compare numbers and dates. Don't ask it to write anything. Don't ask it to predict the real world: one tester asked Clef which thumbnails would beat a channel's average, and its picks did exactly as well as the average, a coin flip. And don't ask for long chains of reasoning in one judgment: an independent test found OpenAI's Decisions confidence no better than chance five reasoning steps deep. ### The rule (3:06) Which leaves one rule, best put in the README of the open model Jeff. Reason in code, decide with the model. Code does the counting, the sums, the dates and the steps. The model makes the judgment that code can't. ### Quick check (3:22) **Quiz:** Flag orders where the customer was charged more than the quote. Which design? A. A Noul: “Was the customer overcharged?” B. Select both amounts from candidates, then compare them in code C. A Score for how overcharged they were D. A Choice between “overcharged” and “fine”, plus “unsure” Answer: B. Comparing numbers is code's job. Use the model to pick which amount is the quote and which is the charge, then compare them exactly. ### Try it (3:34) Try it. Have your agent audit your questions for jagged edges. ```text Audit every decision-model question in this codebase against the known failure modes: counting, arithmetic, number or date comparison, indirection (questions that point at one item in a long list), large irrelevant state, adversarial input, criteria that contradict the instruction, option order, and anything that asks for generation. For each hit, rewrite it so code does the computing and the model only judges meaning, and add a test with a known answer for each rewrite. ``` ### Recap (3:39) So: decision models are jagged. Strong on meaning, weak on counting, arithmetic and dates. Typed output can still be wrong, sometimes confidently. Know the nine failure modes, and reason in code, decide with the model. Next: proving it works, on your own data. ## Sources - TypeSafe AI docs: Jev 1.13 jaggedness. https://docs.typesafe.ai/model-jaggedness/jev-1.13 - The TWIML AI Podcast with Sam Charrington: Asking JEV's Creator the Hard Questions About This Hot New AI Model (at 34:30). https://www.youtube.com/watch?v=9Aato-NfjoU&t=2070s - Flavio Copes: How to build typed decisions with Jev. https://flaviocopes.com/jev/ - Mike Ramos: Jev is the 🐠 at the poker table. https://backnotprop.com/blog/jev-poker/ - Hacker News commenters: Hacker News: Introducing System One Models and Jev. https://news.ycombinator.com/item?id=49717558 - firelex/jeff: firelex/jeff: Millisecond decisions, any domain: a 0.8B open "System 1" model that picks between your…. https://github.com/firelex/jeff - Theo - t3․gg: Jev is incredible (at 13:33). https://www.youtube.com/watch?v=F3YXg7AaKWE&t=813s - Better Stack: Cloudflare Just Challenged Jev, So I Tested Their New Model (Clef) (at 5:55). https://www.youtube.com/watch?v=Uihz1NkhFM0&t=355s - Ryan Porter: The OpenAI Decisions API Needs a Confidence You Can Trust | Anthus. https://anth.us/blog/openai-decisions-api-preview/ - InfoQ: TypeSafe AI Releases Jev: a Decision-Only Model That Returns Typed Probabilities. https://www.infoq.com/news/2026/10/typesafe-ai-jev-released/ - Sebastien Dubois: Jev and System One models - a semantic if statement for your code. https://www.dsebastien.net/jev-and-system-one-models-a-semantic-if-statement-for-your-code/ License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 20: Evaluate on your own data > The only benchmark that matters is the one you ship. Here is the evaluation loop for decision models: real inputs, careful labels, per-question results, enough examples to trust a difference, and a regression test that pins the model version, with what we measured when Jev went from 1.13 to 1.14. Course: System One (https://learn-system-one.pages.dev/) · Module 6: Shipping it · 4 min · Watch: https://learn-system-one.pages.dev/lessons/evaluate-on-your-data/ ## Key takeaways - Evaluate on real inputs from your own logs, labeled by people who know the domain; there is no way around labeling some data. - Thirty examples can reveal your failure modes, but measuring accuracy takes hundreds: at 80% accuracy the margin is about ±14 points with 30 labels and ±4.5 with 300. - Look per question, not just overall: one weak question can hide inside a good average. - Test every change, even wording: in one test, more careful wording made agreement and calibration worse. - Pin the model version; in our tests, Jev 1.14 put 6% fewer answers above 0.99, so the same threshold automated less. ## Practices for coding agents - Build the eval set from real, recent inputs (stratified to include rare cases), labeled by domain experts with a written label guide. - Report accuracy with its margin of error, per question, and include calibration and coverage at your target. - Pin the exact model version and rerun the eval as a regression test on every change to questions, criteria, state or model. - Rerun threshold tuning after any model upgrade before switching traffic. ## Lesson transcript ### Intro (0:01) Every number in this course came from someone else's data, or ours. None of them is your data. This lesson is about getting the only numbers that count. ### Label some data (0:12) Hamel Husain, who consults on AI evals, sees teams resist one step more than any other. There is no getting out of it: you have to label some data. It's the only way to build trust. ### The loop (0:26) Here's the loop. Sample real inputs from your logs, not examples you made up, and include the rare cases on purpose. Have people who know the domain label them, with a written guide, and where two experts disagree, fix the guide first. Run the model and keep every probability. Then measure, fix the biggest failure, and run it again. ### How many labels? (0:49) How many labels? Husain finds that around thirty examples is often when you stop discovering new kinds of failure. But thirty is far too few to measure accuracy. At eighty percent, the margin of error with thirty labels is plus or minus fourteen points. With three hundred, four and a half. With a thousand, two and a half. ### Per question (1:12) Look at each question separately. A Better Stack test compared Jev and Clef on five questions about comments, against a Claude referee. Overall, Jev agreed eighty six percent of the time, Clef eighty three. But Clef won three of the five questions. The whole gap came from one question about tone, where Clef scored forty nine percent. It loves calling things neutral. ### Test every change (1:37) Test every change, even wording. Emil Lindfors had Jev read Norwegian consultation letters about a salmon tax, first with short draft questions, then with careful ones, full of qualifiers about passing mentions and quotes. The careful version was worse. Agreement went down, and calibration error almost tripled, as answers drifted toward the middle. His advice: ask the way you'd ask a colleague across the desk, and keep the fine print in your code. ### Pin the version (2:09) And pin the model version. When Jev one point fourteen came out, we reran both of our tests. Accuracy held, and calibration improved. But on the banking test, the share of answers at point nine nine or above fell from fifty to forty four percent. If your code auto-approves above point nine nine, the upgrade quietly sends six percent more of your traffic to people. Better model, different thresholds. ### Make it a test (2:36) So make your eval a test. Pin the exact model. Keep a few hundred labeled, real inputs in the repository. And fail the build if accuracy, calibration or coverage at your target drops. Run it whenever a question, a criterion, the state or the model changes. ```python MODEL = "jev-1.13.0" # pinned: never "latest" def test_refund_decision(): rows = load_jsonl("evals/refund_300.jsonl") # real, expert-labeled res = run_decisions(MODEL, rows) # keeps every probability assert accuracy(res) >= 0.90 assert ece(res) <= 0.08 assert coverage(res, target=0.95) >= 0.55 # share we can automate ``` ### Quick check (2:55) **Quiz:** On 30 labeled examples, model A scores 87%, model B 83%. Conclusion? A. Model A is better; switch to it B. Almost nothing: that's one example, inside a ±13-point margin C. Model B is better calibrated D. Average the two models Answer: B. The difference is a single example. Thirty labels are good for finding failure modes, not for ranking models: label a few hundred, and compare per question. ### Try it (3:10) Try it. Have your agent turn one of your decisions into a pinned regression eval. ```text For our most important decision-model question, build a regression eval: 1. Sample 300 recent real inputs from logs (stratify so rare classes appear) and export them for expert labeling with a short label guide. 2. Pin the exact model version in one constant. 3. Write a test that reports accuracy with its 95% margin, per-question results, calibration error and coverage at our target, and fails if any drops below the current baseline. 4. Run it in CI on every change to questions, criteria, state or model. ``` ### Recap (3:16) So: label some data, from real inputs, with experts. Thirty labels find failures, hundreds measure them. Look per question. Test every change, even wording. And pin the version, because a better model can still move your thresholds. Next: deciding responsibly. ## Sources - Hamel Husain: LLM Evals: Common Mistakes (at 5:53). https://www.youtube.com/watch?v=GL0XhAj5LPE&t=353s - Lenny's Podcast: Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar. https://www.youtube.com/watch?v=BsWxPI9UM4c - Emil Lindfors: An early-access test of TypeSafe's Jev: calibrated judgments for half a cent. https://lindfors.no/blog/a-first-look-at-typesafes-jev/ - Better Stack: Cloudflare Just Challenged Jev, So I Tested Their New Model (Clef) (at 4:21). https://www.youtube.com/watch?v=Uihz1NkhFM0&t=261s - Ryan Porter: The OpenAI Decisions API Needs a Confidence You Can Trust | Anthus. https://anth.us/blog/openai-decisions-api-preview/ - IndyDevDan: 10 Levels of Jev For Agentic Engineers (at 28:30). https://www.youtube.com/watch?v=_U-O5lYhJ7Q&t=1710s - TypeSafe AI: Lies, Damned Lies, and Benchmarks. https://typesafe.ai/blog/antibenchmaxxing - JevBench authors: JevBench v1 (benchmark for Jev-class typed decision models). https://github.com/doxanocap/jevbench - Connor Frank: An Independent Evaluation of TypeSafe's Jev | Vals AI. https://www.vals.ai/blogs/independent-evaluation-of-jev - Amir Rafe amir rafe edu: Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with…. https://arxiv.org/html/2609.24052 - Hamel Husain: Using LLM-as-a-Judge For Evaluation: A Complete Guide – Hamel’s Blog. https://hamel.dev/blog/posts/llm-judge/ License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 21: Decide responsibly > A decision model makes the same call the same way, millions of times. That removes noise, and it makes any bias systematic. Here is how to audit decisions about people (including the name-swap test we ran), where the reasons come from when the model gives none, and what a responsible deployment logs, samples and leaves to a person. Course: System One (https://learn-system-one.pages.dev/) · Module 6: Shipping it · 4 min · Watch: https://learn-system-one.pages.dev/lessons/responsible-decisions/ ## Key takeaways - Decision models have little noise, so their mistakes are systematic: the same error, for the same kind of case, every time. - Bias can hide in a criterion as well as in a model: the Apgar score's skin-color item gives Black newborns lower scores. - Audit by subgroup and with counterfactual tests; in our name-swap test, answers moved at most 0.06 and no decision flipped, which is one test, not proof of fairness. - The model gives numbers, not reasons; your decomposed questions and thresholds are the reasons, so log them and turn them into explanations. - Keep a person accountable: sample automated decisions for review, give people a way to appeal, and be most careful with decisions about jobs, money, housing and health. ## Practices for coding agents - Before automating decisions about people, run counterfactual tests (swap names and other attributes that must not matter) and compare outcomes by subgroup on labeled data. - Log every decision with input reference, model version, every answer, thresholds and the action taken, and generate explanations from that log. - Sample a share of automated decisions for human review every week, and provide an appeal path to a person. - Use decision models to assist, not decide alone, for hiring, credit, housing and health, and check the rules that apply where you operate. ## Lesson transcript ### Intro (0:01) A decision model will make the same call the same way, a million times a day. That's its strength. It's also why some of those calls deserve extra care. ### Noise and bias (0:12) Olivier Sibony and Daniel Kahneman separate two kinds of error. Noise is scatter: in one audit at an investment firm, two analysts valuing the same security differed by forty four percent on average. Bias is a shared, predictable error. Decision models have very little noise: in our tests, repeated runs moved by four hundredths at most. So whatever errors they have are systematic. The same mistake, for the same kind of case, every time. ### Decisions about people (0:43) That matters most when the decision is about a person. Simon Willison wrote on launch day: I really hope nobody uses Jev to rank job applicants. That floating point number could conceal all manner of unseen bias baked into the models. ### Bias in the criteria (0:59) And bias doesn't only live inside models. It can live in your criteria. Remember the Apgar score from lesson twelve? One of its five items is skin color. A study of more than nine million American newborns found that three point three percent of white babies scored a perfect ten, and only one point four percent of Black babies, which can mean unneeded treatment for healthy infants. ### Test it (1:24) So test it, with counterfactuals. Change only what must not matter, and see if the answer moves. We wrote twelve candidate summaries for a backend job and sent each one five times: with no name, and as Emily Walsh, Greg Baker, Lakisha Washington and Jamal Jones. The average chance of an interview barely moved. No single answer moved by more than point oh six, and no decision flipped. That's reassuring, and it proves very little. It's one test, of one attribute, on twelve made-up people. Bias can travel through proxies a name test never touches: schools, addresses, gaps in a career, the way someone writes. As Willison's advice goes, a model like this needs more evals, not fewer. Test by subgroup on real, labeled cases too. ### Where the reasons come from (2:18) People also ask for reasons. Kahneman pointed out that algorithms clearly better than judges at predicting parole violations still meet resistance, because they can't explain their reasoning. A decision model gives you numbers, not reasons. But your design has them. Unsiloed put it simply: the explanations come from our application. Jev returns the typed answers. So log the reasons you designed. Every decision record holds the model version, every narrow answer, the thresholds and the action. That's an audit trail. And when someone asks why, turn that log into an explanation, with a template or an LLM. The decision stays in your code; only the wording is generated. ```python log.write({ "case": ticket.id, "model": "jev-1.13.0", "at": now(), "answers": {"duplicate_charge": 0.98, "policy_allows": 0.99}, "rule": "refund if both > 0.80", "action": "auto_refund", "review_sample": random() < 0.02, # a person checks 2% }) explain = explain_from(log) # a template or an LLM: wording only ``` ### Keep a person accountable (3:04) And keep a person accountable. Sample automated decisions for review, every week, because a threshold that worked at launch can drift. Give people a way to appeal to a human. Audit outcomes by subgroup. And for decisions about jobs, money, housing or health, let the model assist a person rather than decide alone, and check the rules where you operate. ### Quick check (3:29) **Quiz:** A name-swap test changes no decisions. What can you conclude? A. The model is fair; ship it B. Names alone didn't move these cases; proxies and real subgroups still need testing C. The test was broken D. Bias only matters for LLMs Answer: B. A counterfactual test checks one path for bias. Proxies like schools or addresses, and outcomes on real labeled cases by subgroup, still need checking. ### Try it (3:40) Try it. Have your agent add a decision log and a counterfactual test to one decision that touches people. ```text For one decision in our product that affects a person (approvals, refunds, moderation, screening): 1. Log every decision: input reference, model version, every answer, thresholds, action, and a flag that sends 2% to human review. 2. Generate a plain-language explanation from that log. 3. Write a counterfactual test: for 20 real cases, swap attributes that must not matter (names, pronouns, locations) and report the largest change in each answer and any decision that flips. ``` ### Recap (3:47) So: decision models trade noise for systematic error. Bias can sit in models and in criteria, so test with counterfactuals and by subgroup, and don't treat one pass as proof. Your questions are the reasons: log them. And keep a person accountable. Next, the capstone: putting the whole course together. ## Sources - Simon Willison: Jev introduces a new shape of LLM—System One, aka Decision Models. https://simonwillison.net/2026/Sep/21/jev/ - Sebastien Dubois: Jev and System One models - a semantic if statement for your code. https://www.dsebastien.net/jev-and-system-one-models-a-semantic-if-statement-for-your-code/ - CFA Institute: Olivier Sibony: How can you overcome noise and bias in decision making? (at 8:44). https://www.youtube.com/watch?v=wiYaXOGE_e4&t=524s - Robinson's Podcast Clips: Decision Hygiene for Reducing Bias and Noise in Judgment | Daniel Kahneman & Robinson Erhardt (at 5:27). https://www.youtube.com/watch?v=zt2cFBC8hdw&t=327s - Authority control databases International GND FAST National United States Israel Other Yale LUX: Apgar score - Wikipedia. https://en.wikipedia.org/wiki/Apgar_score - Lex Fridman: Daniel Kahneman: Thinking Fast and Slow, Deep Learning, and AI | Lex Fridman Podcast #65 (at 37:22). https://www.youtube.com/watch?v=UwwBG-MbniY&t=2242s - IBM Technology: Building Decision Agents with LLMs & Machine Learning Models (at 19:50). https://www.youtube.com/watch?v=mRkJTXDromw&t=1190s - Unsiloed AI: Document Review With Unsiloed and TypeSafe Jev. https://www.unsiloed.ai/blog/document-review-unsiloed-typesafe-jev - Hamel Husain: LLM Evals: Common Mistakes (at 26:26). https://www.youtube.com/watch?v=GL0XhAj5LPE&t=1586s - TypeSafe AI docs: Parallel questions. https://docs.typesafe.ai/cookbooks/parallel_questions License: CC BY 4.0 · https://github.com/Haple/system-one --- # Lesson 22: Capstone: a support triage pipeline > The whole course in one pipeline you can run: code prepares the state, one request asks fifteen questions, and code routes every ticket to a refund, a reset link, an LLM, a log, a person or a block. Here is how we built it, the three bad questions our first run exposed, what held-out tickets showed, and why twelve out of twelve still proves less than it sounds. Course: System One (https://learn-system-one.pages.dev/) · Module 6: Shipping it · 5 min · Watch: https://learn-system-one.pages.dev/lessons/capstone-support-triage/ ## Key takeaways - The pipeline: code prepares (redacts cards, lists candidate orders), one request asks 15 questions, code decides and logs. - The model judges meaning; code checks facts: a refund needs the model to hear a duplicate charge and the records to show one, under the $100 limit. - Our first run routed 14 of 24 tickets right; three badly written questions caused every miss. - After fixing them, 24 of 24, and 12 of 12 on tickets written before the fix; 16 of 36 needed no person. - Twelve out of twelve still allows a true error rate up to about 25%: evaluate on hundreds of real tickets before trusting it. ## Practices for coding agents - Split compound questions and describe the company or domain in the state so the model can judge scope. - Write contrastive criteria for any question that a normal request could trip, such as injection checks. - Write held-out test cases before fixing questions, and score fixes only on those. - Require both the model's judgment and an exact code check before any automated action that moves money. ## Lesson transcript ### Intro (0:01) Time to put it all together. One pipeline, built from everything in this course, that triages a support inbox. The code is in the course repository, and you can run it yourself. ### The design (0:13) Here's the design. First, code prepares the state. It redacts card numbers, so no model ever sees one, and lists the customer's orders as candidates. Then one request asks fifteen questions at once, in a quarter of a second. Then code decides: thresholds, policy checks, and a log of every answer. Here's one ticket. My card, four one one one and so on, was charged twice for order P-4410. The card number is already gone. The answers: billing, at one point oh. A duplicate charge, point nine nine. The order, picked from the customer's own orders: P-4410. No injection, no escalation, one issue. ### Meaning and facts (1:02) Now the refund rule. The model must hear a duplicate charge, at point eight or more, the threshold we derived from costs in lesson ten. It must be sure which order. And then code checks the facts: the records really show two identical charges, and the amount is under the hundred dollar limit in the policy. The model judges meaning. Code checks facts. Both have to agree before money moves. ```python if cat == "billing" and n("reports_duplicate") >= ACT: # 0.8 order = find(customer.orders, a["order"].choice) if order is None or a["order"].confidence < 0.9: return "human", "can't tell which order" if not has_identical_charges(order): return "human", "claimed duplicate not in our records" if order.charges[0] > CREDIT_LIMIT_USD: # policy: $100 return "human", "over the credit limit" return "auto_refund", "duplicate confirmed in records" ``` ### Where tickets go (1:30) Every ticket lands in a route. Code handles refunds, reset links, the feature board, and closing thank-yous and spam. An LLM answers how-to questions from the docs. People get everything unclear, risky or over the limit, with escalations, critical bugs and repeat contacts at the front of the line. And attempts to instruct the system are blocked before any LLM sees them. ### The first run (1:54) Our first run was humbling: fourteen of twenty four tickets routed the way we'd labeled them. Error analysis found three bad questions behind every miss. We fixed them, and got twenty four of twenty four. But we'd fixed them while looking at those same tickets, so before rerunning, we wrote twelve new ones. Those came out twelve of twelve. The three bad questions are worth studying, because they're the mistakes you'll make too. Is it in English and about ACME? Two conditions in one question, about a company the state never described: answers hovered around point five. Does it try to give instructions to support? Taken literally, thank you, you can close this ticket, scored point nine five. And a vague line between billing and how to left an invoice question at point five one. Each fix came from an earlier lesson. ### What it proves (2:50) Now apply lesson twenty to our own result. Twelve out of twelve sounds perfect. But with zero errors in twelve tries, the rule of three says the true error rate could still be as high as one in four. These are thirty six tickets we wrote ourselves. Before this pipeline touches a real inbox, it needs a few hundred real tickets, labeled by people who know the policy. ### What it costs (3:14) And what does it cost? Fifteen questions per ticket, about a thousand input tokens, a quarter of a second. That's about four cents per thousand tickets on Jev. Sixteen of our thirty six tickets needed no person at all, and every decision left a complete record behind. ### The whole course (3:33) Look at what went into it. A state with only what the questions need. Typed questions, each about one thing. Fifteen of them in one request. Candidates selected, never generated. Thresholds from costs. Guards in code and in questions. Routes to code, an LLM or a person. A log of every judgment. And an honest eval, held out before the fix. ### Build yours (4:00) Now build yours. Point your coding agent at the capstone in the course repository, and have it adapt the pipeline to your own inbox, with your own policy and your own labeled tickets. ```text Using capstone/triage.py as a template, build a triage pipeline for our support inbox. Put our real policy rules in code where they are exact (limits, windows, plans) and in question criteria where they are about meaning. Keep one request per ticket, card and secret redaction in code, and a JSONL log of every answer. Export 300 recent real tickets for labeling, hold out 100 before tuning any question, and report route accuracy with its margin of error, coverage and cost. ``` ### Recap (4:11) So: code prepares, the model judges, code decides. Facts are checked in code before money moves. Bad questions show up in error analysis, and fixes are scored on held-out cases. And twelve for twelve is a smoke test, not proof. That's the capstone. ### The end (4:32) That's the course. Software has always needed decisions, and now it can make the judgments that used to need a person, in a fraction of a second, for a fraction of a cent, with honest probabilities. Use them where they're strong, check them where they're weak, and keep a person close to the decisions that matter. Thanks for watching. ## Sources - TypeSafe AI docs: Speculative fan-out. https://docs.typesafe.ai/patterns/fan-out - TypeSafe AI docs: Intent routing. https://docs.typesafe.ai/patterns/intent-routing - TypeSafe AI docs: Guardrails for LLMs. https://docs.typesafe.ai/cookbooks/llm_guardrails - TypeSafe AI docs: Jev 1.13 jaggedness. https://docs.typesafe.ai/model-jaggedness/jev-1.13 - TypeSafe AI docs: Pre-parsed value extraction. https://docs.typesafe.ai/cookbooks/pre_parsed_value_extraction_cookbook - TypeSafe AI: TypeSafe agent skill (SKILL.md). https://github.com/typesafe-ai/skills/blob/main/skills/typesafe-ai/SKILL.md License: CC BY 4.0 · https://github.com/Haple/system-one