A language model writes its answer one token at a time. A decision model reads your state once and scores every allowed answer in a single pass. Here is the core trick in a few lines of Python, what production models add on top (trained readout heads, shared state, calibration training), and why typed output is not the same as a correct answer.
Key takeaways
- An LLM's answer costs one sequential pass per output token; a decision model scores all allowed answers in one pass.
- The core trick: read the model's probability for each allowed option instead of generating text, then normalize.
- Production decision models add a trained readout head, shared state across questions, and training aimed at calibrated probabilities.
- Because it only scores the options you supplied, it can never return a value outside your schema.
- Typed is not the same as correct: a decision model can still be confidently wrong. That is why probabilities and calibration matter.
Try it with your agent
Paste this into your coding agent (needs Python):
Build tiny_decider.py: given a text and a list of options, put the options in a prompt with letters A, B, C..., run a small open GGUF model (e.g. Qwen3-0.6B via llama-cpp-python) once, and return a probability for each option from the next-token logits (softmax over the letters). Then run it on 20 real examples from this project and print the probabilities next to the labels we expect.
Transcript
Intro
What's actually inside a decision model? Less magic than you'd think. In this lesson, you'll see the core trick in a few lines of code, and what the real models add on top.
Generate vs score
Start with a normal language model. It writes its answer one token at a time, and every token is another full pass through the network. A fifty token answer is fifty passes, one after another. A decision model never writes. It reads your state once, and scores every allowed answer in a single pass.
The core trick
Here's the core trick, adapted from a popular post called Jev in twenty five lines of Python. Put the options in the prompt with letters. Run a small open model once. Then, instead of letting it write, read the probability it gives each letter as the next token, and normalize.
On a tiny model, that phishing email comes out about eighty nine percent phishing, eight percent spam, and three percent legitimate. A distribution over exactly your options. Nothing to parse, and no way to answer outside the list.
What real models add
Real decision models start from that idea and add three layers. A trained readout head, instead of guessing letter tokens. A shared state, read once, so many questions cost almost nothing extra. And training aimed at one thing: probabilities you can trust.
AWS's open Strands Decider is a good look inside. They take a two billion parameter model, throw away the part that generates words, and add a tiny pointer head of about a million parameters that compares each option to the question. One forward pass, no generation, no decoding loop.
Three training goals
The training goal is where these models differ most from chatbots. Reinforcement learning from human feedback, which made ChatGPT, rewards answers people prefer. Verifiable rewards made reasoning models good at math and code. TypeSafe calls its third path reinforcement learning for calibrated decisions: reward probabilities that match how often the answer turns out right.
Typed is not correct
Now, the big claim at launch was that Jev can't hallucinate. Hacker News pushed back fast. As one commenter put it, it can still forward a billing query to the dev department. Even TypeSafe's founder agreed in the thread: these models are probabilistic, so they can be confidently wrong.
So what you get is a probability, and the real promise is that the probability is honest. When a decision model says point nine, it should be right about nine times out of ten. Whether that holds on your data is something you'll learn to measure in module three.
Quick check
Quick check. Why can a decision model never return an answer outside your list?
Try it
Try it. Ask your coding agent to build the tiny decider from this lesson with any small open model, and compare it with a hosted decision model on twenty of your own examples.
Recap
So: decision models read once and score every allowed answer in one pass. The trick is reading probabilities instead of writing text. Real models add a trained head, shared state, and calibration training. And typed doesn't mean correct. Next: the four decision models you can use today.