{
 "title": "System One",
 "url": "https://learn-system-one.pages.dev",
 "description": "Learn to build with System One decision models like TypeSafe's Jev, OpenAI's Decisions API, Cloudflare's Clef and AWS's Strands Decider. 22 short animated lessons on typed questions, calibrated probabilities, thresholds, composition patterns and evaluation, distilled from the docs, 57 talks and 70+ articles and papers, plus original measurements. Free and open source, and your agent can take it too.",
 "license": "CC-BY-4.0",
 "modules": [
  {
   "title": "The idea",
   "lessons": [
    {
     "number": 1,
     "title": "Software needs decisions, not essays",
     "url": "https://learn-system-one.pages.dev/lessons/software-needs-decisions/",
     "markdown": "https://learn-system-one.pages.dev/lessons/software-needs-decisions.md",
     "minutes": 4,
     "summary": "Language models write beautifully, but most software needs a decision it can branch on. Here is the prompt-and-parse problem, the new shape of a decision model (state and typed questions in, typed answers and probabilities out), and how a whole category of these models appeared in three weeks.",
     "takeaways": [
      "Most AI calls inside software are really decisions: which team, is it urgent, is this allowed.",
      "Prompt-and-parse (ask an LLM for JSON, then parse and retry) works, but it is slow, fragile and gives you no trustworthy confidence.",
      "A decision model takes state plus typed questions and returns typed answers with probabilities. It cannot return a value you did not offer.",
      "Code keeps control of the workflow; the decision model answers narrow questions; hard cases go to a reasoning model or a person.",
      "In three weeks of September and October 2026, TypeSafe, OpenAI, Cloudflare and AWS all shipped models of this kind."
     ],
     "practices": [
      "When an LLM call exists only to produce a label, a yes/no or a rating that code branches on, consider replacing it with a typed question to a decision model.",
      "Keep control flow, rules and side effects in code; use the model only for the judgment code cannot express.",
      "Never parse free text to recover a decision when a typed answer with probabilities is available."
     ]
    },
    {
     "number": 2,
     "title": "System One and System Two",
     "url": "https://learn-system-one.pages.dev/lessons/system-one-and-system-two/",
     "markdown": "https://learn-system-one.pages.dev/lessons/system-one-and-system-two.md",
     "minutes": 3,
     "summary": "The name comes from Daniel Kahneman: fast, automatic judgment versus slow, effortful reasoning. Here is what the metaphor gets right about decision models, where it breaks, and the rule of thumb it gives you: if an expert could answer in a few seconds, it is a System One question.",
     "takeaways": [
      "Kahneman's System 1 is fast, automatic and effortless; System 2 is slow, serial and effortful. Decision models aim at the first kind of judgment.",
      "Rule of thumb: if a knowledgeable person could answer in a few seconds with the right context, it is a System One question.",
      "Tasks that need several steps of reasoning, new text or planning are System Two work: send them to a reasoning model, a person, or decompose them.",
      "The useful architecture is a layer cake: code is exact, the decision model judges, the language model thinks.",
      "The two systems are a metaphor, and fast does not have to mean sloppy: what makes fast judgments usable is honest uncertainty."
     ],
     "practices": [
      "Before calling a decision model, check that each question is a few-second judgment for an expert given the state; if not, split it or route it to a reasoning model.",
      "Keep exact computation (math, dates, counting, lookups) in code, never in a fast judgment.",
      "Escalate System Two work (multi-step reasoning, writing, planning) to an LLM or a person instead of forcing it into one question."
     ]
    },
    {
     "number": 3,
     "title": "How a decision model works",
     "url": "https://learn-system-one.pages.dev/lessons/how-decision-models-work/",
     "markdown": "https://learn-system-one.pages.dev/lessons/how-decision-models-work.md",
     "minutes": 4,
     "summary": "A language model writes its answer one token at a time. A decision model reads your state once and scores every allowed answer in a single pass. Here is the core trick in a few lines of Python, what production models add on top (trained readout heads, shared state, calibration training), and why typed output is not the same as a correct answer.",
     "takeaways": [
      "An LLM's answer costs one sequential pass per output token; a decision model scores all allowed answers in one pass.",
      "The core trick: read the model's probability for each allowed option instead of generating text, then normalize.",
      "Production decision models add a trained readout head, shared state across questions, and training aimed at calibrated probabilities.",
      "Because it only scores the options you supplied, it can never return a value outside your schema.",
      "Typed is not the same as correct: a decision model can still be confidently wrong. That is why probabilities and calibration matter."
     ],
     "practices": [
      "Treat a decision model's output as a probability distribution over your options, not as a fact.",
      "Never assume 'cannot return an invalid value' means 'cannot be wrong'; gate risky actions on probability and test on your data.",
      "Keep the set of allowed answers complete, including a none-of-the-above option when inputs can fall outside it."
     ]
    },
    {
     "number": 4,
     "title": "Jev, Decisions, Clef, Strands: the landscape",
     "url": "https://learn-system-one.pages.dev/lessons/decision-models-compared/",
     "markdown": "https://learn-system-one.pages.dev/lessons/decision-models-compared.md",
     "minutes": 4,
     "summary": "Four decision models you can use today, one request shape they almost share, and what they cost. Then what we measured ourselves: accuracy on a public benchmark and on fresh news published after every model's release, latency from a real server, and why you should trust your own evals over anyone's leaderboard.",
     "takeaways": [
      "Jev (TypeSafe, hosted), OpenAI's Decisions API (hosted), Clef (Cloudflare, open weights) and Strands Decider (AWS, open, runs locally) share one shape: state plus typed questions in, probabilities out.",
      "Jev, Clef and Strands speak the same System One request format; OpenAI's is the same idea with different field names.",
      "Pricing is per input token with free output: a thousand decisions typically cost cents.",
      "Public benchmarks can leak into training data and leaderboards move; vendor latency is model time, not your round trip.",
      "Choose on your constraints (data residency, images, latency from your region, price, fine-tuning) and then measure on your own data."
     ],
     "practices": [
      "Hide the vendor behind a small adapter so you can swap decision models and compare them on the same questions.",
      "Pin a versioned model ID in production and log it with every decision.",
      "Measure latency from where your code runs, and accuracy and calibration on your own labeled examples, before choosing a model."
     ]
    }
   ]
  },
  {
   "title": "Asking good questions",
   "lessons": [
    {
     "number": 5,
     "title": "Noul: the probability of yes",
     "url": "https://learn-system-one.pages.dev/lessons/noul-yes-no-questions/",
     "markdown": "https://learn-system-one.pages.dev/lessons/noul-yes-no-questions.md",
     "minutes": 4,
     "summary": "The simplest decision is yes or no. A Noul returns the probability that the answer is yes, and that single number carries both the answer and the doubt. Here is how to read it, where to put thresholds, why 0.5 never means medium, and how to phrase yes/no questions the model can't misread.",
     "takeaways": [
      "A Noul returns one number: the probability that the answer is yes. Near 1 is a strong yes, near 0 a strong no, near 0.5 means it can't tell.",
      "0.5 is not 'medium'. If you need a degree, use a Score with described levels.",
      "Thresholds live in your code and depend on the cost of each mistake; values in the middle can go to a person.",
      "One condition per question, phrased so a high value means yes, with the exact condition spelled out.",
      "Short, plain questions beat long qualified ones; keep the fine print in your code."
     ],
     "practices": [
      "Use a Noul for a clean yes/no condition; use a Score when the answer is a degree.",
      "Write one condition per Noul and phrase it so a high probability means yes; avoid negations.",
      "Threshold Noul values in code, with a review band in the middle sized by the cost of errors.",
      "Add true/false criteria when the boundary is subtle, and keep questions short and literal."
     ]
    },
    {
     "number": 6,
     "title": "Choice: one answer from a set",
     "url": "https://learn-system-one.pages.dev/lessons/choice-questions/",
     "markdown": "https://learn-system-one.pages.dev/lessons/choice-questions.md",
     "minutes": 4,
     "summary": "A Choice picks one option from a set you define and returns a probability for every option. Here is how to read the distribution and its confidence, why a none-of-the-above option matters, how contrastive descriptions sharpen close calls, and what reversing the option order did to 3,080 real answers.",
     "takeaways": [
      "A Choice returns the chosen option, a probability for every option, and a confidence that summarizes how peaked that distribution is.",
      "A split distribution is information: a ticket that belongs to two teams should look split.",
      "Always offer a none-of-the-above option when inputs can fall outside your list; otherwise the model must pick something.",
      "Option names and descriptions are both sent to the model; for close calls, describe what each option covers and what it doesn't.",
      "Option order can sway uncertain answers: in our test, reversing 77 options changed 4.9% of answers, none above 0.9 probability."
     ],
     "practices": [
      "Give every Choice the full list of options with short descriptions, plus an explicit other / none option when coverage isn't guaranteed.",
      "Use contrastive descriptions (what it covers, what it is not for, examples) for options that get confused.",
      "Treat low-confidence Choice answers as candidates for review, and re-check close calls with the options reordered."
     ]
    },
    {
     "number": 7,
     "title": "Score: a position on a scale you describe",
     "url": "https://learn-system-one.pages.dev/lessons/score-questions/",
     "markdown": "https://learn-system-one.pages.dev/lessons/score-questions.md",
     "minutes": 4,
     "summary": "A Score places the state on ordered levels you describe and returns a position that can fall between them. Here is how the number is computed, why levels must describe situations rather than degrees, why one Score should measure one thing, and how an extra top level separates the dangerous cases.",
     "takeaways": [
      "A Score returns a position along your levels: each level number times its probability, added up. It can land between levels.",
      "Describe situations, not degrees: 'workaround exists' gives the model something to match; '1' or 'moderate' doesn't.",
      "The same score can hide different shapes; read the probabilities and confidence, not just the number.",
      "One dimension per Score; split compound scales and combine the parts in code.",
      "Give a rare extreme case its own top level so it doesn't hide inside 'very bad'."
     ],
     "practices": [
      "Write every Score level as a concrete, self-contained situation; never use bare numbers or relative words like 'worse'.",
      "Keep each Score to one dimension; normalize scores (divide by the top level) before weighting them together.",
      "Add a separate top level for rare extremes that need different handling (abuse, outages, safety).",
      "Use the score to rank or threshold, not to reconstruct an exact quantity between levels."
     ]
    },
    {
     "number": 8,
     "title": "State: give it what it needs",
     "url": "https://learn-system-one.pages.dev/lessons/designing-state/",
     "markdown": "https://learn-system-one.pages.dev/lessons/designing-state.md",
     "minutes": 4,
     "summary": "The state is everything the model sees besides your questions. Here is how to shape it (named fields, paths the questions point at, the record next to the policy), what we measured when we buried the one thing that mattered, and the limits on length, language and context you should design around.",
     "takeaways": [
      "State is the material you would hand an expert panel before asking: the message, the record, the policy.",
      "Use a JSON object with named fields, and point each question at the field it is about with a backticked path.",
      "Unrelated text in its own named field barely hurt in our test; asking about one item buried by index among 40 similar ones dropped accuracy from 98% to 72%.",
      "Name the thing you are asking about: one item per request, or put the record in the question itself.",
      "Filter in code before you send; long context exists but is slower and less accurate, and English is still the strongest language."
     ],
     "practices": [
      "Send state as a JSON object with descriptive field names; reference fields in questions with backticked paths like ticket.messages[0].text.",
      "Put the decision's evidence (policy, record, history) in the state, and only what the questions need.",
      "Never make the model find one item by index in a long list of similar items; send that item on its own or inside the question.",
      "Write the complete question in instructions; question IDs are not sent to the model."
     ]
    }
   ]
  },
  {
   "title": "Probabilities you can act on",
   "lessons": [
    {
     "number": 9,
     "title": "Calibration: does 0.9 mean 90%?",
     "url": "https://learn-system-one.pages.dev/lessons/calibration/",
     "markdown": "https://learn-system-one.pages.dev/lessons/calibration.md",
     "minutes": 4,
     "summary": "A probability is only useful if it means what it says. Here is what calibration is, how to see it in a reliability diagram, why chat-trained models drift into overconfidence, what independent tests and our own measurements found for decision models, and how a few hundred labels let you measure and fix it.",
     "takeaways": [
      "Calibrated means: of all the answers given probability 0.8, about 80% turn out right.",
      "A reliability diagram plots stated probability against how often the model was right; the diagonal is perfect.",
      "Post-training for chat tends to make models overconfident; decision models are trained to keep probabilities honest, but that is a claim to test.",
      "Calibration is a property of each model on each task: in our tests Jev was slightly overconfident at the top, and on fresh news an LLM's stated confidence was better calibrated.",
      "Measure it on a few hundred of your own labels (ECE, Brier score), and fix systematic bias with simple recalibration such as Platt scaling."
     ],
     "practices": [
      "Before relying on probabilities, measure calibration on a labeled sample from your own traffic: a reliability table, ECE and Brier score.",
      "Never assume a vendor's calibration claim holds on your data; re-check after every model version change.",
      "If probabilities are systematically off, fit a recalibration map (Platt scaling or isotonic regression) on held-out labels; it fixes calibration but cannot add information.",
      "Prefer native probabilities to numbers a model writes as text, but measure both if you use both."
     ]
    },
    {
     "number": 10,
     "title": "Confidence and thresholds",
     "url": "https://learn-system-one.pages.dev/lessons/confidence-thresholds/",
     "markdown": "https://learn-system-one.pages.dev/lessons/confidence-thresholds.md",
     "minutes": 4,
     "summary": "The answer tells you what; the probability tells you whether to act. Here is the three-path pattern (act, confirm, escalate), why thresholds should differ by action, a one-line formula that turns the cost of mistakes into a threshold, and the one thing thresholds cannot protect you from.",
     "takeaways": [
      "Use probability as a second axis: the answer says what to do, the probability says whether to do it now.",
      "Three paths: act automatically, ask to confirm, or escalate to a person or a slower model.",
      "Thresholds belong to actions, not to the model: a read-only action can go at 0.6, a money transfer may need 0.85 and a confirmation.",
      "With calibrated probabilities, act when p exceeds cost of a false action ÷ (cost of a false action + cost of a missed one).",
      "Confidence can't flag inputs the model doesn't understand at all; check that inputs are in scope before you trust the number."
     ],
     "practices": [
      "Gate every automated action on its own threshold, set by the cost of acting wrongly versus missing a true case.",
      "Keep thresholds as named constants in code with a comment explaining their cost reasoning, and log the probability with each action.",
      "Send the middle band to confirmation or human review instead of guessing.",
      "Detect out-of-scope inputs (wrong language, unknown domain) before the decision, because confidence will not catch them."
     ]
    },
    {
     "number": 11,
     "title": "Automate the sure ones",
     "url": "https://learn-system-one.pages.dev/lessons/selective-automation/",
     "markdown": "https://learn-system-one.pages.dev/lessons/selective-automation.md",
     "minutes": 4,
     "summary": "You don't need a model that is right about everything. You need to know which answers you can trust. Here is the coverage curve that answers 'how much can we automate at 95%?', how to pick thresholds from a few hundred labels, what confidence-ordered review saves on our own data, and how to answer less precisely instead of wrongly.",
     "takeaways": [
      "The real question is: what share of decisions can we automate at the accuracy we need?",
      "A coverage curve answers it: on our banking data, Jev was 80% right overall, but 96% right on the half it was surest about.",
      "Pick thresholds from 200 to 500 labeled examples by simulating each cutoff; treat the result as an estimate.",
      "Reviewing the least confident answers first reached 95% system accuracy with 33% human review, versus 75% for random review.",
      "When unsure, answer less precisely: falling back to a broader category turned 40% right into 70% right in TypeSafe's SEC example."
     ],
     "practices": [
      "Build a coverage table (threshold, share automated, accuracy) from labeled examples before choosing a threshold.",
      "Route the least confident answers to people first, and size the review queue from the coverage table.",
      "When a taxonomy is hierarchical, return the parent category instead of a guess when confidence is low.",
      "For rare positive classes, judge thresholds by precision and recall on the flagged items, not overall accuracy."
     ]
    }
   ]
  },
  {
   "title": "Composing judgments",
   "lessons": [
    {
     "number": 12,
     "title": "Decompose: many small questions",
     "url": "https://learn-system-one.pages.dev/lessons/decompose-judgments/",
     "markdown": "https://learn-system-one.pages.dev/lessons/decompose-judgments.md",
     "minutes": 4,
     "summary": "TypeSafe calls it the most important concept in its guide: break one broad judgment into narrow questions and combine the answers in code. Here is what decomposition buys you (reasons, control, per-signal thresholds), why decision science has recommended it for seventy years, and the case where it made things worse.",
     "takeaways": [
      "A broad question hides several judgments behind one number; narrow questions expose them so you can inspect, tune and combine them.",
      "The broad answer is often right, but only the narrow ones tell you why: which signal fired, which tool argument was wrong.",
      "Combine in code: weighted sums for factors that trade off, separate hard rules for 'any serious violation'.",
      "Decision science agrees: simple formulas over separately judged criteria beat holistic judgment (Meehl, Dawes, Kahneman, the Apgar score).",
      "Decomposition is not magic: it can't turn a System Two problem into System One answers, so validate the combination on labeled data."
     ],
     "practices": [
      "Split broad judgments into narrow, independent questions, each a few-second call for an expert; ask them in one request.",
      "Combine answers in code with explicit weights or rules; keep the raw per-question answers for debugging and explanations.",
      "Use separate conditions for veto rules instead of letting a weighted sum average a serious violation away.",
      "Check on labeled data that the decomposed version actually beats the single question before switching."
     ]
    },
    {
     "number": 13,
     "title": "Ask everything at once",
     "url": "https://learn-system-one.pages.dev/lessons/speculative-fan-out/",
     "markdown": "https://learn-system-one.pages.dev/lessons/speculative-fan-out.md",
     "minutes": 3,
     "summary": "Questions about the same state belong in one request: they are evaluated in parallel and independently, so extra questions add a few tokens and almost no time. Here is what we measured with 13 questions, why you should ask speculative questions you might not need, and the few cases where a second request is genuinely warranted.",
     "takeaways": [
      "Every question in a request sees the same state and is evaluated independently and in parallel.",
      "In our test, 13 questions in one request took 0.27 seconds and 1,828 tokens; 13 separate requests took 3.3 seconds and 18,940 tokens, with the same answers.",
      "Ask speculative questions: everything your code might need, including ones that only matter on some branches, and let code ignore the rest.",
      "A second request is warranted only when an earlier answer changes the state or the options of the next question.",
      "Coding agents tend to write one question per call; check their integrations for this."
     ],
     "practices": [
      "Batch every question about the same state into one request, including speculative ones for later branches.",
      "Split into a second request only when the next question's state or options depend on an earlier answer.",
      "Review agent-written integrations for one-question-per-call loops and merge them."
     ]
    },
    {
     "number": 14,
     "title": "Route and cascade",
     "url": "https://learn-system-one.pages.dev/lessons/routing-and-cascades/",
     "markdown": "https://learn-system-one.pages.dev/lessons/routing-and-cascades.md",
     "minutes": 4,
     "summary": "One fast decision in front of expensive things. Here is intent routing (code, a specialist model, or a person), the cascade (cheap model first, escalate only the uncertain), how to tune it so it matches a strong model at a fraction of the cost, and the prompt-cache gotcha that turned one developer's model router into a money loser.",
     "takeaways": [
      "Routing: one cheap, fast decision chooses the handler, which may be plain code, a specialist LLM, or a person.",
      "If you can map the whole decision tree, build it explicitly and optimize each node; don't hand it to an agent loop.",
      "A cascade answers with the cheap model when it's confident and escalates the rest to a stronger model or a person.",
      "Tune the cascade on a sample of a few hundred items: pick the cheapest threshold that still agrees with the strong model at your target rate.",
      "Route per task, not mid-session: switching models mid-conversation throws away the prompt cache."
     ],
     "practices": [
      "Put one fast decision (intent + complexity) in front of expensive handlers, and send low-confidence routes to a person.",
      "Implement cascades: accept confident cheap answers, escalate the rest, and tune the threshold on labeled data against the strong model's answers.",
      "Prefer routing whole tasks or reasoning effort over swapping models mid-session, which invalidates prompt caches."
     ]
    },
    {
     "number": 15,
     "title": "Select, don't generate",
     "url": "https://learn-system-one.pages.dev/lessons/select-dont-generate/",
     "markdown": "https://learn-system-one.pages.dev/lessons/select-dont-generate.md",
     "minutes": 3,
     "summary": "A decision model can't write a value, but most extraction is really selection: the value is already in the text. Let code find the candidates and let the model pick one, then copy it verbatim. Here is that pattern for phone numbers, dates, search reranking, line-level search and deep taxonomies.",
     "takeaways": [
      "Most extraction is selection: the value already exists in the source, so the model only has to pick it.",
      "Code finds candidates (regex, parser, search), the model chooses, code copies the value verbatim, so nothing is invented.",
      "Dates are parts: month, day and year are each a small Choice with a 'not stated' option; code assembles and does the arithmetic.",
      "Reranking is selection too: a cheap search finds candidates, one Noul per candidate scores relevance.",
      "Big taxonomies go level by level, keeping the best few paths instead of one greedy guess."
     ],
     "practices": [
      "For extraction, generate candidates in code and ask a Choice which candidate is the requested value; always include a none option.",
      "Extract dates as separate Choices for each part with 'not stated'; build and compare dates in code.",
      "Rerank search results with one relevance question per candidate and sort by probability.",
      "Walk hierarchical taxonomies level by level with a small beam of top paths."
     ]
    },
    {
     "number": 16,
     "title": "Verify and guard",
     "url": "https://learn-system-one.pages.dev/lessons/verification-and-guardrails/",
     "markdown": "https://learn-system-one.pages.dev/lessons/verification-and-guardrails.md",
     "minutes": 4,
     "summary": "Decision models are cheap and fast enough to check everything: every citation an LLM writes, every message going in and out, every extracted field. Here is a citation check, a one-request guardrail screen, why two independent checks beat one, and why none of this is a security boundary.",
     "takeaways": [
      "Use decision models as fast judges: check claims against sources, screen messages, sanity-check extracted values.",
      "One request can screen a message for several hazards plus a severity score, mapped by code to pass, review, block or route.",
      "Your policy lives in your code, consistent across model versions, instead of in each lab's moving refusal line.",
      "When code can check something exactly (a verbatim quote, a sum), let code do it.",
      "A classifier guard reduces risk but is not a security boundary; with private data, untrusted input and outbound access, remove a capability."
     ],
     "practices": [
      "Check LLM citations with a Choice (supports / partially / contradicts / unrelated) against the source context, and verify verbatim quotes with string matching in code.",
      "Screen inputs and outputs with one request of narrow hazard questions plus a severity Score; map them to pass, review, block or route in code.",
      "Combine independent checks (extraction confidence and plausibility) and keep both reasons when both fail.",
      "Never rely on a decision-model guard alone to stop prompt injection; limit the agent's capabilities instead."
     ]
    }
   ]
  },
  {
   "title": "Real systems",
   "lessons": [
    {
     "number": 17,
     "title": "Decision models inside agents",
     "url": "https://learn-system-one.pages.dev/lessons/decision-models-in-agents/",
     "markdown": "https://learn-system-one.pages.dev/lessons/decision-models-in-agents.md",
     "minutes": 4,
     "summary": "Agents make dozens of small decisions per task: which tool, which skill, are these arguments grounded, is it time to ask the user, is the task done. Here is where a fast decision model fits in an agent loop, a gate that stops an agent from guessing, what skill selection gained in TypeSafe's test, and real-time control at ten decisions a second.",
     "takeaways": [
      "Most of an agent's decisions are rote; let a decision model make those and save the LLM for the hard ones.",
      "Good spots: model routing, tool and skill selection, argument checks before a tool runs, guardrails, and 'is it done?' checks.",
      "A before-tool-call gate with two narrow questions can make an agent ask instead of guessing.",
      "Selecting skills with a decision model cut wrong skill loads from 16.8% to 7.3% in TypeSafe's test of 182 skills.",
      "Prefer workflows (code-defined paths with smart nodes) when you can map the decision tree; use agents when you can't."
     ],
     "practices": [
      "Gate tool calls on narrow questions (are the arguments grounded in what the user said? is it premature to act?) and guide the agent back instead of hard-failing.",
      "Use a decision model to pick the few relevant tools or skills to load per step, instead of loading everything into context.",
      "Give long-running agents a cheap 'ask the decision model' tool for yes/no and classification checks over files or pages.",
      "Set a deadline for real-time decisions with a deterministic fallback when the answer is late."
     ]
    },
    {
     "number": 18,
     "title": "Judgments as data",
     "url": "https://learn-system-one.pages.dev/lessons/judgments-as-data/",
     "markdown": "https://learn-system-one.pages.dev/lessons/judgments-as-data.md",
     "minutes": 4,
     "summary": "At a fraction of a cent per answer, judgments stop being calls you make and become data you keep: columns you can sort, filter, weight and train on. Here is a study that coded half a million police narratives for $25, a loop that turned tasting notes into features for a classic model, and why cheaper judgment will mean much more of it.",
     "takeaways": [
      "Ask once, store the answers, and reuse them: changing a weight or a filter doesn't need another model call.",
      "Whole corpora are now affordable: a study screened 499,500 police narratives for $25 and coded 195,857 with 27 questions for $30.",
      "Judgments make great features: questions turned wine notes into 67 numeric columns, and a classic model beat every baseline.",
      "With labels, let the data choose the questions: an autoresearch loop proposes questions and keeps the ones that cut error.",
      "Cheaper judgment means more judgment, which is why TypeSafe named Jev after Jevons."
     ],
     "practices": [
      "Store raw per-question answers (probabilities, model ID, date) as data, and recompute composites in code when weights change.",
      "Before coding a large corpus, run a lean screening schema first and the full schema only where it applies.",
      "When you have labels, use decision-model answers as features for a classic model, and keep only questions that improve held-out error.",
      "Estimate cost per 1,000 items from measured token counts before launching a big run."
     ]
    }
   ]
  },
  {
   "title": "Shipping it",
   "lessons": [
    {
     "number": 19,
     "title": "Know the jagged edges",
     "url": "https://learn-system-one.pages.dev/lessons/jagged-edges/",
     "markdown": "https://learn-system-one.pages.dev/lessons/jagged-edges.md",
     "minutes": 4,
     "summary": "Decision models are strong at semantic judgment and unreliable at things a calculator does. Here are the failure modes we reproduced (counting, arithmetic, dates), TypeSafe's own list of nine, the cases where the model is wrong without sounding unsure, and the rule that fixes most of them: reason in code, decide with the model.",
     "takeaways": [
      "Decision models are jagged: excellent at meaning, unreliable at counting, arithmetic and date comparison.",
      "In our tests, one counting question got 5 fruits instead of 6; twelve per-item Nouls summed in code got it right.",
      "On a wrong invoice total and a late payment, the model hedged at 0.43 and 0.47: honest, but code answers exactly.",
      "Typed output can't be malformed, but it can still be wrong, sometimes with confidence.",
      "Don't ask for counts, sums, date math, forecasts of real-world outcomes, or long reasoning chains; reason in code and decide with the model."
     ],
     "practices": [
      "Never ask a decision model to count, add, compare numbers or compare dates; extract the parts and compute in code.",
      "For semantic counting, ask one Noul per item and sum in code.",
      "Write the exact condition in the instructions; when you find yourself explaining what you meant, add that to the question.",
      "Test each new question on known hard cases (negations, look-alike options, adversarial text) before shipping."
     ]
    },
    {
     "number": 20,
     "title": "Evaluate on your own data",
     "url": "https://learn-system-one.pages.dev/lessons/evaluate-on-your-data/",
     "markdown": "https://learn-system-one.pages.dev/lessons/evaluate-on-your-data.md",
     "minutes": 4,
     "summary": "The only benchmark that matters is the one you ship. Here is the evaluation loop for decision models: real inputs, careful labels, per-question results, enough examples to trust a difference, and a regression test that pins the model version, with what we measured when Jev went from 1.13 to 1.14.",
     "takeaways": [
      "Evaluate on real inputs from your own logs, labeled by people who know the domain; there is no way around labeling some data.",
      "Thirty examples can reveal your failure modes, but measuring accuracy takes hundreds: at 80% accuracy the margin is about ±14 points with 30 labels and ±4.5 with 300.",
      "Look per question, not just overall: one weak question can hide inside a good average.",
      "Test every change, even wording: in one test, more careful wording made agreement and calibration worse.",
      "Pin the model version; in our tests, Jev 1.14 put 6% fewer answers above 0.99, so the same threshold automated less."
     ],
     "practices": [
      "Build the eval set from real, recent inputs (stratified to include rare cases), labeled by domain experts with a written label guide.",
      "Report accuracy with its margin of error, per question, and include calibration and coverage at your target.",
      "Pin the exact model version and rerun the eval as a regression test on every change to questions, criteria, state or model.",
      "Rerun threshold tuning after any model upgrade before switching traffic."
     ]
    },
    {
     "number": 21,
     "title": "Decide responsibly",
     "url": "https://learn-system-one.pages.dev/lessons/responsible-decisions/",
     "markdown": "https://learn-system-one.pages.dev/lessons/responsible-decisions.md",
     "minutes": 4,
     "summary": "A decision model makes the same call the same way, millions of times. That removes noise, and it makes any bias systematic. Here is how to audit decisions about people (including the name-swap test we ran), where the reasons come from when the model gives none, and what a responsible deployment logs, samples and leaves to a person.",
     "takeaways": [
      "Decision models have little noise, so their mistakes are systematic: the same error, for the same kind of case, every time.",
      "Bias can hide in a criterion as well as in a model: the Apgar score's skin-color item gives Black newborns lower scores.",
      "Audit by subgroup and with counterfactual tests; in our name-swap test, answers moved at most 0.06 and no decision flipped, which is one test, not proof of fairness.",
      "The model gives numbers, not reasons; your decomposed questions and thresholds are the reasons, so log them and turn them into explanations.",
      "Keep a person accountable: sample automated decisions for review, give people a way to appeal, and be most careful with decisions about jobs, money, housing and health."
     ],
     "practices": [
      "Before automating decisions about people, run counterfactual tests (swap names and other attributes that must not matter) and compare outcomes by subgroup on labeled data.",
      "Log every decision with input reference, model version, every answer, thresholds and the action taken, and generate explanations from that log.",
      "Sample a share of automated decisions for human review every week, and provide an appeal path to a person.",
      "Use decision models to assist, not decide alone, for hiring, credit, housing and health, and check the rules that apply where you operate."
     ]
    },
    {
     "number": 22,
     "title": "Capstone: a support triage pipeline",
     "url": "https://learn-system-one.pages.dev/lessons/capstone-support-triage/",
     "markdown": "https://learn-system-one.pages.dev/lessons/capstone-support-triage.md",
     "minutes": 5,
     "summary": "The whole course in one pipeline you can run: code prepares the state, one request asks fifteen questions, and code routes every ticket to a refund, a reset link, an LLM, a log, a person or a block. Here is how we built it, the three bad questions our first run exposed, what held-out tickets showed, and why twelve out of twelve still proves less than it sounds.",
     "takeaways": [
      "The pipeline: code prepares (redacts cards, lists candidate orders), one request asks 15 questions, code decides and logs.",
      "The model judges meaning; code checks facts: a refund needs the model to hear a duplicate charge and the records to show one, under the $100 limit.",
      "Our first run routed 14 of 24 tickets right; three badly written questions caused every miss.",
      "After fixing them, 24 of 24, and 12 of 12 on tickets written before the fix; 16 of 36 needed no person.",
      "Twelve out of twelve still allows a true error rate up to about 25%: evaluate on hundreds of real tickets before trusting it."
     ],
     "practices": [
      "Split compound questions and describe the company or domain in the state so the model can judge scope.",
      "Write contrastive criteria for any question that a normal request could trip, such as injection checks.",
      "Write held-out test cases before fixing questions, and score fixes only on those.",
      "Require both the model's judgment and an exact code check before any automated action that moves money."
     ]
    }
   ]
  }
 ]
}