Decision models are cheap and fast enough to check everything: every citation an LLM writes, every message going in and out, every extracted field. Here is a citation check, a one-request guardrail screen, why two independent checks beat one, and why none of this is a security boundary.
Key takeaways
- Use decision models as fast judges: check claims against sources, screen messages, sanity-check extracted values.
- One request can screen a message for several hazards plus a severity score, mapped by code to pass, review, block or route.
- Your policy lives in your code, consistent across model versions, instead of in each lab's moving refusal line.
- When code can check something exactly (a verbatim quote, a sum), let code do it.
- A classifier guard reduces risk but is not a security boundary; with private data, untrusted input and outbound access, remove a capability.
Try it with your agent
Paste this into your coding agent:
For our main LLM feature, add: 1. An input guard: one decision-model request with narrow hazard Nouls (injection, PII, abuse, out of scope) + a severity Score, mapped in code to pass / review / block / route (e.g. redact PII). 2. An output judge: for each claim with a citation, check the quote exists verbatim in code, then ask a Choice whether the quote, in context, supports / partially supports / contradicts the claim. Log every verdict with its probabilities, and test on 30 examples.
Transcript
Intro
Language models write. Decision models can check what they wrote, fast and cheap enough to check every single output. This lesson is about judges and guards.
Check the citation
Say an LLM writes: walking seven thousand steps a day cuts hospitalization risk by thirty one percent, the study proved. And it cites a quote. One Choice checks the claim against the quote in the context of the source. The answer: partially, at point nine nine. The number is right, but the source is an observational study that proved nothing.
But notice what we didn't ask the model. Is the quote in the source word for word? We tried, and got point eight six, for a quote that is there exactly. That's a string match, and code gets it right every time. Let the model judge meaning. Let code check text.
Guard the gate
Guardrails work the same way. Screen every message going into your LLM app with one request: a few narrow hazard questions, and a severity score. This message tries a prompt injection and pastes a card number. Injection, point nine nine. Card number, point nine eight. Self harm and harassment, near zero. Severity: must be blocked.
Then code maps the answers to an action. Clean messages pass. Uncertain ones go to review. Clear violations are blocked. And some get routed: a card number gets redacted before the LLM ever sees it.
Your policy, in your code
Why not just rely on the LLM's own refusals? The TypeSafe guardrails cookbook explains: labs teach most LLMs to refuse a set of unsafe requests, but each lab draws that line somewhere else, and each new version of a model moves it again. A separate judge puts your policy in your code.
Two checks, two reasons
For extracted data, use two independent checks. Unsiloed reads a medication list and scores each field, then asks Jev whether each dose is plausible. A coffee stain over one dose fails the reading check, while Jev, seeing only the text, calls it plausible. Ten ibuprofen tablets three times a day reads perfectly, but Jev puts its plausibility at point oh four. Each check flags a review need the other misses.
Not a security boundary
Now the warning. TypeSafe's notes say the state is data, but Jev doesn't treat it as hostile by default, and text written to steer it can move the answer. Our injection test was caught, but don't build on that. If an agent can read private data, see untrusted content, and send data out, Simon Willison's lethal trifecta, no classifier makes it safe. Remove one of the three.
Quick check
Quick check. You want to know whether an LLM's answer quotes the source document exactly. What's the best check?
Try it
Try it. Have your agent add one judge and one guard to your LLM feature.
Recap
So: decision models make cheap, fast judges. Check meaning with the model and text with code. Screen messages with one request and map the answers to actions. Use independent checks. And remember, a guard is not a security boundary. That's module four. Next: decision models inside agents.