{"id":"20-evaluate","number":20,"slug":"evaluate-on-your-data","title":"Evaluate on your own data","short":"Evaluate","module":"Shipping it","summary":"The only benchmark that matters is the one you ship. Here is the evaluation loop for decision models: real inputs, careful labels, per-question results, enough examples to trust a difference, and a regression test that pins the model version, with what we measured when Jev went from 1.13 to 1.14.","voice":"Jessica","scenes":[{"template":"title","chapter":"Intro","narration":"Every number in this course came from someone else's data, or ours. None of them is your data. This lesson is about getting the only numbers that count.","params":{"kicker":"Lesson 20","module":"Module 6 · Shipping it","title":"Evaluate on *your own data*","subtitle":"The only benchmark that matters is the one you ship.","cast":["you","snap"]},"start":0.5,"dur":11.388,"words":[{"text":"Every","s":0.8,"e":1.12},{"text":"number","s":1.28,"e":1.44},{"text":"in","s":1.48,"e":1.52},{"text":"this","s":1.6,"e":1.68},{"text":"course","s":1.76,"e":2.16},{"text":"came","s":2.24,"e":2.4},{"text":"from","s":2.48,"e":2.56},{"text":"someone","s":2.72,"e":2.96},{"text":"else's","s":3.12,"e":3.44},{"text":"data,","s":3.52,"e":4.16},{"text":"or","s":4.28,"e":4.4},{"text":"ours.","s":4.56,"e":5.28},{"text":"None","s":5.52,"e":5.68},{"text":"of","s":5.76,"e":5.84},{"text":"them","s":5.96,"e":6.08},{"text":"is","s":6.16,"e":6.32},{"text":"your","s":6.48,"e":6.72},{"text":"data.","s":6.8,"e":7.2},{"text":"This","s":7.68,"e":7.92},{"text":"lesson","s":8.12,"e":8.32},{"text":"is","s":8.4,"e":8.48},{"text":"about","s":8.6,"e":8.72},{"text":"getting","s":8.8,"e":9.04},{"text":"the","s":9.08,"e":9.12},{"text":"only","s":9.36,"e":9.6},{"text":"numbers","s":9.68,"e":10.08},{"text":"that","s":10.16,"e":10.24},{"text":"count.","s":10.32,"e":11.04}]},{"template":"quote","chapter":"Label some data","narration":"Hamel Husain, who consults on AI evals, sees teams resist one step more than any other. There is no getting out of it: you have to label some data. It's the only way to build trust.","params":{"text":"There is no getting out of: *you have to label some data*.","who":"Hamel Husain","role":"AI evals consultant","src":"LLM Evals: Common Mistakes","t":353},"start":11.888,"dur":13.619,"words":[{"text":"Hamel","s":12.188,"e":12.508},{"text":"Husain,","s":12.548,"e":13.228},{"text":"who","s":13.308,"e":13.388},{"text":"consults","s":13.588,"e":13.828},{"text":"on","s":13.868,"e":14.028},{"text":"AI","s":14.108,"e":14.348},{"text":"evals,","s":14.428,"e":15.228},{"text":"sees","s":15.308,"e":15.548},{"text":"teams","s":15.628,"e":15.868},{"text":"resist","s":15.948,"e":16.348},{"text":"one","s":16.588,"e":16.748},{"text":"step","s":16.828,"e":17.068},{"text":"more","s":17.148,"e":17.388},{"text":"than","s":17.468,"e":17.708},{"text":"any","s":17.828,"e":17.948},{"text":"other.","s":18.028,"e":18.828},{"text":"There","s":18.908,"e":18.988},{"text":"is","s":19.068,"e":19.388},{"text":"no","s":19.468,"e":19.548},{"text":"getting","s":19.628,"e":19.868},{"text":"out","s":19.948,"e":20.028},{"text":"of","s":20.068,"e":20.108},{"text":"it:","s":20.188,"e":20.668},{"text":"you","s":20.748,"e":20.908},{"text":"have","s":21.068,"e":21.308},{"text":"to","s":21.388,"e":21.468},{"text":"label","s":21.668,"e":21.868},{"text":"some","s":21.948,"e":22.028},{"text":"data.","s":22.108,"e":22.828},{"text":"It's","s":22.868,"e":22.988},{"text":"the","s":23.108,"e":23.228},{"text":"only","s":23.308,"e":23.548},{"text":"way","s":23.628,"e":23.788},{"text":"to","s":23.828,"e":23.868},{"text":"build","s":23.988,"e":24.108},{"text":"trust.","s":24.388,"e":24.908}]},{"template":"flow","chapter":"The loop","narration":"Here's the loop. Sample real inputs from your logs, not examples you made up, and include the rare cases on purpose. Have people who know the domain label them, with a written guide, and where two experts disagree, fix the guide first. Run the model and keep every probability. Then measure, fix the biggest failure, and run it again.","params":{"title":"The evaluation loop","nodes":[{"label":"Sample real inputs","sub":"from logs, rare cases included","kind":"curly","edge":"inputs","at":"Sample real inputs"},{"label":"Experts label","sub":"a written guide; resolve disagreement","kind":"you","edge":"labels","at":"know the domain"},{"label":"Run and measure","sub":"per question, keep probabilities","kind":"snap","at":"Run the model"}],"caption":"Fix the biggest failure, then run it again.","caption_at":"fix the biggest"},"start":25.507,"dur":23.232,"words":[{"text":"Here's","s":25.807,"e":26.127},{"text":"the","s":26.167,"e":26.207},{"text":"loop.","s":26.287,"e":26.887},{"text":"Sample","s":27.167,"e":27.487},{"text":"real","s":27.647,"e":27.887},{"text":"inputs","s":27.967,"e":28.367},{"text":"from","s":28.447,"e":28.527},{"text":"your","s":28.567,"e":28.607},{"text":"logs,","s":28.687,"e":29.407},{"text":"not","s":29.487,"e":29.727},{"text":"examples","s":30.007,"e":30.287},{"text":"you","s":30.327,"e":30.367},{"text":"made","s":30.447,"e":30.527},{"text":"up,","s":30.607,"e":31.167},{"text":"and","s":31.207,"e":31.247},{"text":"include","s":31.327,"e":31.727},{"text":"the","s":31.767,"e":31.807},{"text":"rare","s":31.887,"e":32.127},{"text":"cases","s":32.207,"e":32.687},{"text":"on","s":32.807,"e":32.927},{"text":"purpose.","s":33.007,"e":34.047},{"text":"Have","s":34.167,"e":34.287},{"text":"people","s":34.447,"e":34.607},{"text":"who","s":34.687,"e":34.767},{"text":"know","s":34.847,"e":35.007},{"text":"the","s":35.087,"e":35.167},{"text":"domain","s":35.367,"e":35.567},{"text":"label","s":35.647,"e":35.967},{"text":"them,","s":36.087,"e":36.447},{"text":"with","s":36.527,"e":36.607},{"text":"a","s":36.647,"e":36.687},{"text":"written","s":36.767,"e":37.007},{"text":"guide,","s":37.087,"e":37.727},{"text":"and","s":37.767,"e":37.807},{"text":"where","s":37.927,"e":38.047},{"text":"two","s":38.127,"e":38.207},{"text":"experts","s":38.287,"e":38.687},{"text":"disagree,","s":38.767,"e":39.647},{"text":"fix","s":39.807,"e":39.967},{"text":"the","s":40.047,"e":40.127},{"text":"guide","s":40.287,"e":40.447},{"text":"first.","s":40.607,"e":41.567},{"text":"Run","s":41.647,"e":41.727},{"text":"the","s":41.807,"e":41.847},{"text":"model","s":41.887,"e":42.287},{"text":"and","s":42.367,"e":42.447},{"text":"keep","s":42.527,"e":42.767},{"text":"every","s":42.847,"e":43.167},{"text":"probability.","s":43.247,"e":44.367},{"text":"Then","s":44.447,"e":44.607},{"text":"measure,","s":44.687,"e":45.407},{"text":"fix","s":45.487,"e":45.727},{"text":"the","s":45.807,"e":45.967},{"text":"biggest","s":46.047,"e":46.287},{"text":"failure,","s":46.367,"e":46.967},{"text":"and","s":47.007,"e":47.167},{"text":"run","s":47.207,"e":47.247},{"text":"it","s":47.327,"e":47.407},{"text":"again.","s":47.607,"e":48.127}]},{"template":"stat","chapter":"How many labels?","narration":"How many labels? Husain finds that around thirty examples is often when you stop discovering new kinds of failure. But thirty is far too few to measure accuracy. At eighty percent, the margin of error with thirty labels is plus or minus fourteen points. With three hundred, four and a half. With a thousand, two and a half.","params":{"title":"95% margin of error when accuracy is 80%","items":[{"value":14,"prefix":"±","suffix":" pts","label":"30 labels","tone":"bad","at":"thirty labels"},{"value":4.5,"decimals":1,"prefix":"±","suffix":" pts","label":"300 labels","tone":"s2","at":"three hundred"},{"value":2.5,"decimals":1,"prefix":"±","suffix":" pts","label":"1,000 labels","tone":"good","at":"a thousand"}],"src":"1.96 × √(p(1−p)/n), with p = 0.8. Thirty examples find failure modes; hundreds measure them.","punch":false},"start":48.739,"dur":22.997,"words":[{"text":"How","s":49.039,"e":49.279},{"text":"many","s":49.359,"e":49.519},{"text":"labels?","s":49.599,"e":50.439},{"text":"Husain","s":50.719,"e":51.199},{"text":"finds","s":51.359,"e":51.599},{"text":"that","s":51.639,"e":51.679},{"text":"around","s":51.759,"e":51.999},{"text":"thirty","s":52.239,"e":52.479},{"text":"examples","s":52.559,"e":53.199},{"text":"is","s":53.279,"e":53.519},{"text":"often","s":53.599,"e":53.839},{"text":"when","s":53.919,"e":53.999},{"text":"you","s":54.079,"e":54.159},{"text":"stop","s":54.239,"e":54.479},{"text":"discovering","s":54.559,"e":55.119},{"text":"new","s":55.199,"e":55.359},{"text":"kinds","s":55.439,"e":55.679},{"text":"of","s":55.719,"e":55.759},{"text":"failure.","s":55.839,"e":56.799},{"text":"But","s":56.919,"e":57.039},{"text":"thirty","s":57.199,"e":57.519},{"text":"is","s":57.639,"e":57.759},{"text":"far","s":57.919,"e":58.159},{"text":"too","s":58.239,"e":58.399},{"text":"few","s":58.559,"e":58.719},{"text":"to","s":58.799,"e":58.879},{"text":"measure","s":58.959,"e":59.199},{"text":"accuracy.","s":59.559,"e":60.479},{"text":"At","s":60.599,"e":60.719},{"text":"eighty","s":60.799,"e":61.039},{"text":"percent,","s":61.119,"e":61.759},{"text":"the","s":61.839,"e":61.919},{"text":"margin","s":62.079,"e":62.239},{"text":"of","s":62.319,"e":62.399},{"text":"error","s":62.479,"e":62.719},{"text":"with","s":62.839,"e":62.959},{"text":"thirty","s":63.039,"e":63.279},{"text":"labels","s":63.519,"e":63.759},{"text":"is","s":63.839,"e":63.999},{"text":"plus","s":64.079,"e":64.319},{"text":"or","s":64.359,"e":64.399},{"text":"minus","s":64.479,"e":64.799},{"text":"fourteen","s":64.959,"e":65.359},{"text":"points.","s":65.519,"e":66.399},{"text":"With","s":66.479,"e":66.559},{"text":"three","s":66.639,"e":66.719},{"text":"hundred,","s":66.879,"e":67.519},{"text":"four","s":67.599,"e":67.679},{"text":"and","s":67.719,"e":67.759},{"text":"a","s":67.839,"e":67.879},{"text":"half.","s":67.919,"e":68.719},{"text":"With","s":68.799,"e":68.879},{"text":"a","s":68.959,"e":69.039},{"text":"thousand,","s":69.119,"e":70.079},{"text":"two","s":70.159,"e":70.239},{"text":"and","s":70.319,"e":70.399},{"text":"a","s":70.439,"e":70.479},{"text":"half.","s":70.679,"e":71.119}]},{"template":"statement","chapter":"Per question","narration":"Look at each question separately. A Better Stack test compared Jev and Clef on five questions about comments, against a Claude referee. Overall, Jev agreed eighty six percent of the time, Clef eighty three. But Clef won three of the five questions. The whole gap came from one question about tone, where Clef scored forty nine percent. It loves calling things neutral.","params":{"eyebrow":"Better Stack, Jev vs Clef on five questions","text":"86% vs 83% overall. But *one question* held the whole gap.","sub":"Clef won 3 of 5 questions, and scored 49% on tone.","sub_at":"won three of the five","art":"chart","tone":"s2","punch":"forty nine percent"},"start":71.736,"dur":25.401,"words":[{"text":"Look","s":72.036,"e":72.276},{"text":"at","s":72.316,"e":72.356},{"text":"each","s":72.516,"e":72.676},{"text":"question","s":72.756,"e":73.156},{"text":"separately.","s":73.316,"e":73.956},{"text":"A","s":74.436,"e":74.516},{"text":"Better","s":74.596,"e":74.756},{"text":"Stack","s":74.836,"e":75.156},{"text":"test","s":75.236,"e":75.476},{"text":"compared","s":75.556,"e":75.876},{"text":"Jev","s":75.956,"e":76.276},{"text":"and","s":76.316,"e":76.356},{"text":"Clef","s":76.516,"e":76.836},{"text":"on","s":76.956,"e":77.076},{"text":"five","s":77.156,"e":77.396},{"text":"questions","s":77.476,"e":77.876},{"text":"about","s":77.956,"e":78.116},{"text":"comments,","s":78.196,"e":78.756},{"text":"against","s":78.916,"e":79.236},{"text":"a","s":79.276,"e":79.316},{"text":"Claude","s":79.396,"e":79.636},{"text":"referee.","s":79.716,"e":80.196},{"text":"Overall,","s":80.676,"e":81.396},{"text":"Jev","s":81.476,"e":81.716},{"text":"agreed","s":81.796,"e":82.116},{"text":"eighty","s":82.196,"e":82.436},{"text":"six","s":82.596,"e":82.756},{"text":"percent","s":82.836,"e":83.076},{"text":"of","s":83.156,"e":83.196},{"text":"the","s":83.236,"e":83.316},{"text":"time,","s":83.476,"e":83.956},{"text":"Clef","s":84.036,"e":84.356},{"text":"eighty","s":84.596,"e":84.756},{"text":"three.","s":84.836,"e":85.236},{"text":"But","s":85.716,"e":85.796},{"text":"Clef","s":85.876,"e":86.196},{"text":"won","s":86.276,"e":86.516},{"text":"three","s":86.676,"e":86.996},{"text":"of","s":87.076,"e":87.156},{"text":"the","s":87.196,"e":87.236},{"text":"five","s":87.316,"e":87.476},{"text":"questions.","s":87.556,"e":88.436},{"text":"The","s":88.676,"e":88.756},{"text":"whole","s":88.836,"e":88.996},{"text":"gap","s":89.076,"e":89.396},{"text":"came","s":89.476,"e":89.556},{"text":"from","s":89.636,"e":89.876},{"text":"one","s":89.956,"e":90.116},{"text":"question","s":90.196,"e":90.596},{"text":"about","s":90.716,"e":90.836},{"text":"tone,","s":90.996,"e":91.636},{"text":"where","s":91.716,"e":91.796},{"text":"Clef","s":91.876,"e":92.116},{"text":"scored","s":92.196,"e":92.516},{"text":"forty","s":92.676,"e":92.916},{"text":"nine","s":92.996,"e":93.156},{"text":"percent.","s":93.236,"e":94.116},{"text":"It","s":94.196,"e":94.356},{"text":"loves","s":94.516,"e":94.916},{"text":"calling","s":95.076,"e":95.236},{"text":"things","s":95.316,"e":95.556},{"text":"neutral.","s":95.716,"e":96.516}]},{"template":"table","chapter":"Test every change","narration":"Test every change, even wording. Emil Lindfors had Jev read Norwegian consultation letters about a salmon tax, first with short draft questions, then with careful ones, full of qualifiers about passing mentions and quotes. The careful version was worse. Agreement went down, and calibration error almost tripled, as answers drifted toward the middle. His advice: ask the way you'd ask a colleague across the desk, and keep the fine print in your code.","params":{"title":"Same questions, two wordings (192 judgments)","cols":["","Draft wording","Careful wording"],"widths":"1.4fr 1fr 1fr","rows":[{"cells":["Agreement with reference","0.89","0.86"],"at":"Agreement went down"},{"cells":["Calibration error (ECE)","*0.040*","0.116"],"hl":true,"at":"almost tripled"},{"cells":["Answers between 0.3 and 0.7","24","41"],"at":"toward the middle"}],"note":"Emil Lindfors, an early-access test of TypeSafe's Jev (24 documents, jev-1.13.0).","note_at":"colleague across the desk"},"start":97.137,"dur":31.957,"words":[{"text":"Test","s":97.437,"e":97.757},{"text":"every","s":97.997,"e":98.237},{"text":"change,","s":98.397,"e":99.117},{"text":"even","s":99.277,"e":99.437},{"text":"wording.","s":99.517,"e":100.157},{"text":"Emil","s":100.717,"e":101.037},{"text":"Lindfors","s":101.117,"e":101.677},{"text":"had","s":101.757,"e":101.917},{"text":"Jev","s":101.997,"e":102.237},{"text":"read","s":102.317,"e":102.477},{"text":"Norwegian","s":102.557,"e":103.037},{"text":"consultation","s":103.117,"e":103.757},{"text":"letters","s":103.837,"e":104.157},{"text":"about","s":104.237,"e":104.397},{"text":"a","s":104.477,"e":104.557},{"text":"salmon","s":104.677,"e":104.877},{"text":"tax,","s":104.957,"e":105.837},{"text":"first","s":105.917,"e":106.237},{"text":"with","s":106.317,"e":106.397},{"text":"short","s":106.477,"e":106.717},{"text":"draft","s":106.877,"e":107.037},{"text":"questions,","s":107.117,"e":107.997},{"text":"then","s":108.077,"e":108.317},{"text":"with","s":108.397,"e":108.477},{"text":"careful","s":108.637,"e":109.037},{"text":"ones,","s":109.117,"e":109.677},{"text":"full","s":109.757,"e":109.997},{"text":"of","s":110.037,"e":110.077},{"text":"qualifiers","s":110.237,"e":110.877},{"text":"about","s":110.997,"e":111.117},{"text":"passing","s":111.197,"e":111.597},{"text":"mentions","s":111.877,"e":112.157},{"text":"and","s":112.197,"e":112.237},{"text":"quotes.","s":112.397,"e":113.157},{"text":"The","s":113.437,"e":113.517},{"text":"careful","s":113.597,"e":113.997},{"text":"version","s":114.077,"e":114.557},{"text":"was","s":114.717,"e":114.877},{"text":"worse.","s":115.037,"e":115.717},{"text":"Agreement","s":115.917,"e":116.397},{"text":"went","s":116.517,"e":116.637},{"text":"down,","s":116.717,"e":117.197},{"text":"and","s":117.277,"e":117.357},{"text":"calibration","s":117.437,"e":117.997},{"text":"error","s":118.077,"e":118.397},{"text":"almost","s":118.597,"e":118.797},{"text":"tripled,","s":118.877,"e":119.677},{"text":"as","s":119.757,"e":119.837},{"text":"answers","s":119.917,"e":120.317},{"text":"drifted","s":120.397,"e":120.797},{"text":"toward","s":120.877,"e":121.037},{"text":"the","s":121.117,"e":121.157},{"text":"middle.","s":121.197,"e":121.797},{"text":"His","s":122.077,"e":122.237},{"text":"advice:","s":122.317,"e":123.037},{"text":"ask","s":123.277,"e":123.517},{"text":"the","s":123.597,"e":123.637},{"text":"way","s":123.677,"e":123.837},{"text":"you'd","s":123.917,"e":124.077},{"text":"ask","s":124.197,"e":124.317},{"text":"a","s":124.357,"e":124.397},{"text":"colleague","s":124.477,"e":124.797},{"text":"across","s":124.877,"e":125.197},{"text":"the","s":125.237,"e":125.277},{"text":"desk,","s":125.357,"e":125.997},{"text":"and","s":126.077,"e":126.317},{"text":"keep","s":126.557,"e":126.717},{"text":"the","s":126.797,"e":126.877},{"text":"fine","s":126.957,"e":127.197},{"text":"print","s":127.277,"e":127.517},{"text":"in","s":127.557,"e":127.597},{"text":"your","s":127.677,"e":127.757},{"text":"code.","s":127.837,"e":128.477}]},{"template":"table","chapter":"Pin the version","narration":"And pin the model version. When Jev one point fourteen came out, we reran both of our tests. Accuracy held, and calibration improved. But on the banking test, the share of answers at point nine nine or above fell from fifty to forty four percent. If your code auto-approves above point nine nine, the upgrade quietly sends six percent more of your traffic to people. Better model, different thresholds.","params":{"title":"BANKING77, 3,080 messages, same questions","cols":["","jev-1.13.0","jev-1.14.0"],"widths":"1.5fr 1fr 1fr","rows":[{"cells":["Accuracy","80.1%","80.3%"],"at":"Accuracy held"},{"cells":["Calibration error (ECE)","0.089","0.071"],"at":"calibration improved"},{"cells":["Answers ≥ 0.99","50.3%","*44.4%*"],"hl":true,"at":"fifty to forty four"},{"cells":["Accuracy of those answers","96.3%","96.4%"]}],"note":"Measured 2026-10-15. On our fresh news test, answers ≥ 0.99 fell from 54.5% to 49.5%.","note_at":"Better model"},"start":129.094,"dur":27.308,"words":[{"text":"And","s":129.394,"e":129.634},{"text":"pin","s":129.714,"e":129.954},{"text":"the","s":130.034,"e":130.114},{"text":"model","s":130.194,"e":130.434},{"text":"version.","s":130.514,"e":131.274},{"text":"When","s":131.554,"e":131.714},{"text":"Jev","s":131.794,"e":132.034},{"text":"one","s":132.114,"e":132.194},{"text":"point","s":132.274,"e":132.514},{"text":"fourteen","s":132.634,"e":132.994},{"text":"came","s":133.074,"e":133.234},{"text":"out,","s":133.314,"e":133.634},{"text":"we","s":133.754,"e":133.874},{"text":"reran","s":133.994,"e":134.354},{"text":"both","s":134.434,"e":134.594},{"text":"of","s":134.674,"e":134.714},{"text":"our","s":134.754,"e":134.834},{"text":"tests.","s":134.914,"e":135.594},{"text":"Accuracy","s":135.794,"e":136.354},{"text":"held,","s":136.554,"e":136.834},{"text":"and","s":136.954,"e":137.074},{"text":"calibration","s":137.154,"e":137.794},{"text":"improved.","s":137.874,"e":138.754},{"text":"But","s":138.994,"e":139.074},{"text":"on","s":139.114,"e":139.154},{"text":"the","s":139.234,"e":139.314},{"text":"banking","s":139.394,"e":139.714},{"text":"test,","s":139.794,"e":140.354},{"text":"the","s":140.434,"e":140.514},{"text":"share","s":140.634,"e":140.754},{"text":"of","s":140.834,"e":140.914},{"text":"answers","s":141.114,"e":141.314},{"text":"at","s":141.394,"e":141.554},{"text":"point","s":141.674,"e":141.794},{"text":"nine","s":141.874,"e":142.114},{"text":"nine","s":142.194,"e":142.274},{"text":"or","s":142.354,"e":142.434},{"text":"above","s":142.514,"e":142.994},{"text":"fell","s":143.154,"e":143.314},{"text":"from","s":143.394,"e":143.634},{"text":"fifty","s":143.714,"e":144.114},{"text":"to","s":144.274,"e":144.514},{"text":"forty","s":144.634,"e":144.834},{"text":"four","s":144.914,"e":145.074},{"text":"percent.","s":145.154,"e":146.194},{"text":"If","s":146.274,"e":146.354},{"text":"your","s":146.434,"e":146.514},{"text":"code","s":146.594,"e":146.834},{"text":"auto-approves","s":146.914,"e":147.634},{"text":"above","s":147.834,"e":148.034},{"text":"point","s":148.114,"e":148.274},{"text":"nine","s":148.354,"e":148.594},{"text":"nine,","s":148.714,"e":149.154},{"text":"the","s":149.234,"e":149.314},{"text":"upgrade","s":149.514,"e":149.714},{"text":"quietly","s":149.874,"e":150.354},{"text":"sends","s":150.434,"e":150.834},{"text":"six","s":150.954,"e":151.074},{"text":"percent","s":151.154,"e":151.554},{"text":"more","s":151.634,"e":151.794},{"text":"of","s":151.834,"e":151.874},{"text":"your","s":151.954,"e":152.034},{"text":"traffic","s":152.114,"e":152.434},{"text":"to","s":152.514,"e":152.594},{"text":"people.","s":152.674,"e":153.314},{"text":"Better","s":153.554,"e":153.714},{"text":"model,","s":153.794,"e":154.514},{"text":"different","s":154.674,"e":154.834},{"text":"thresholds.","s":155.114,"e":155.794}]},{"template":"code","chapter":"Make it a test","narration":"So make your eval a test. Pin the exact model. Keep a few hundred labeled, real inputs in the repository. And fail the build if accuracy, calibration or coverage at your target drops. Run it whenever a question, a criterion, the state or the model changes.","params":{"file":"test_refund_decision.py","lang":"py","lines":["MODEL = \"jev-1.13.0\"                 # pinned: never \"latest\"","","def test_refund_decision():","    rows = load_jsonl(\"evals/refund_300.jsonl\")   # real, expert-labeled","    res = run_decisions(MODEL, rows)             # keeps every probability","    assert accuracy(res) >= 0.90","    assert ece(res) <= 0.08","    assert coverage(res, target=0.95) >= 0.55     # share we can automate"],"hl":[{"from":1,"to":1,"label":"pin the version","tone":"agent","at":"Pin the exact model"},{"from":6,"to":8,"label":"fail the build on regressions","tone":"bad","at":"fail the build"}]},"start":156.402,"dur":19.001,"words":[{"text":"So","s":156.702,"e":156.942},{"text":"make","s":157.102,"e":157.262},{"text":"your","s":157.342,"e":157.422},{"text":"eval","s":157.502,"e":157.902},{"text":"a","s":157.982,"e":158.062},{"text":"test.","s":158.142,"e":158.702},{"text":"Pin","s":159.102,"e":159.262},{"text":"the","s":159.342,"e":159.422},{"text":"exact","s":159.502,"e":160.062},{"text":"model.","s":160.142,"e":160.542},{"text":"Keep","s":160.942,"e":161.022},{"text":"a","s":161.102,"e":161.182},{"text":"few","s":161.262,"e":161.342},{"text":"hundred","s":161.422,"e":161.742},{"text":"labeled,","s":161.822,"e":162.462},{"text":"real","s":162.622,"e":162.782},{"text":"inputs","s":162.862,"e":163.262},{"text":"in","s":163.302,"e":163.342},{"text":"the","s":163.422,"e":163.689},{"text":"repository.","s":163.955,"e":164.302},{"text":"And","s":164.702,"e":164.782},{"text":"fail","s":164.942,"e":165.182},{"text":"the","s":165.222,"e":165.262},{"text":"build","s":165.342,"e":165.662},{"text":"if","s":165.742,"e":165.822},{"text":"accuracy,","s":165.902,"e":166.702},{"text":"calibration","s":166.782,"e":167.422},{"text":"or","s":167.502,"e":167.582},{"text":"coverage","s":167.742,"e":168.142},{"text":"at","s":168.182,"e":168.222},{"text":"your","s":168.302,"e":168.382},{"text":"target","s":168.462,"e":168.782},{"text":"drops.","s":168.942,"e":169.502},{"text":"Run","s":169.902,"e":170.062},{"text":"it","s":170.142,"e":170.222},{"text":"whenever","s":170.262,"e":170.622},{"text":"a","s":170.702,"e":170.782},{"text":"question,","s":170.862,"e":171.502},{"text":"a","s":171.529,"e":171.555},{"text":"criterion,","s":171.582,"e":172.462},{"text":"the","s":172.502,"e":172.542},{"text":"state","s":172.702,"e":173.022},{"text":"or","s":173.102,"e":173.262},{"text":"the","s":173.342,"e":173.422},{"text":"model","s":173.502,"e":173.822},{"text":"changes.","s":173.902,"e":174.782}]},{"template":"quiz","chapter":"Quick check","narration":"Quick check. On thirty labeled examples, model A scores eighty seven percent and model B eighty three. What can you conclude?","params":{"q":"On **30** labeled examples, model A scores **87%**, model B **83%**. Conclusion?","options":["Model A is better; switch to it","Almost nothing: that's one example, inside a ±13-point margin","Model B is better calibrated","Average the two models"],"answer":1,"explain":"The difference is a single example. Thirty labels are good for finding failure modes, not for ranking models: label a few hundred, and compare per question."},"start":175.403,"dur":14.329,"words":[{"text":"Quick","s":175.703,"e":175.943},{"text":"check.","s":176.023,"e":176.543},{"text":"On","s":176.743,"e":176.903},{"text":"thirty","s":177.063,"e":177.303},{"text":"labeled","s":177.383,"e":177.703},{"text":"examples,","s":178.023,"e":178.583},{"text":"model","s":178.663,"e":178.903},{"text":"A","s":179.063,"e":179.303},{"text":"scores","s":179.383,"e":179.783},{"text":"eighty","s":179.863,"e":180.023},{"text":"seven","s":180.183,"e":180.343},{"text":"percent","s":180.423,"e":180.903},{"text":"and","s":181.063,"e":181.223},{"text":"model","s":181.303,"e":181.543},{"text":"B","s":181.703,"e":181.943},{"text":"eighty","s":182.103,"e":182.343},{"text":"three.","s":182.423,"e":183.223},{"text":"What","s":183.303,"e":183.463},{"text":"can","s":183.543,"e":183.623},{"text":"you","s":183.663,"e":183.703},{"text":"conclude?","s":183.783,"e":184.583}]},{"template":"exercise","chapter":"Try it","narration":"Try it. Have your agent turn one of your decisions into a pinned regression eval.","params":{"title":"Build a regression eval","intro":"Paste this into your coding agent:","prompt":"For our most important decision-model question, build a regression eval:\n1. Sample 300 recent real inputs from logs (stratify so rare classes\n   appear) and export them for expert labeling with a short label guide.\n2. Pin the exact model version in one constant.\n3. Write a test that reports accuracy with its 95% margin, per-question\n   results, calibration error and coverage at our target, and fails if\n   any drops below the current baseline.\n4. Run it in CI on every change to questions, criteria, state or model."},"start":189.732,"dur":5.939,"words":[{"text":"Try","s":190.032,"e":190.272},{"text":"it.","s":190.352,"e":190.592},{"text":"Have","s":190.912,"e":191.072},{"text":"your","s":191.152,"e":191.232},{"text":"agent","s":191.392,"e":191.552},{"text":"turn","s":191.632,"e":191.872},{"text":"one","s":191.952,"e":192.032},{"text":"of","s":192.112,"e":192.152},{"text":"your","s":192.192,"e":192.272},{"text":"decisions","s":192.352,"e":192.912},{"text":"into","s":192.992,"e":193.152},{"text":"a","s":193.232,"e":193.312},{"text":"pinned","s":193.392,"e":193.712},{"text":"regression","s":193.832,"e":194.272},{"text":"eval.","s":194.352,"e":195.072}]},{"template":"recap","chapter":"Recap","narration":"So: label some data, from real inputs, with experts. Thirty labels find failures, hundreds measure them. Look per question. Test every change, even wording. And pin the version, because a better model can still move your thresholds. Next: deciding responsibly.","params":{"items":["Real inputs, expert labels: no way around it.","30 labels find failures; hundreds measure them.","Per question; every change, even wording.","Pin the version; rerun before you upgrade."],"next":"Decide responsibly","next_at":"Next"},"start":195.671,"dur":21.544,"words":[{"text":"So:","s":195.971,"e":196.771},{"text":"label","s":196.851,"e":197.171},{"text":"some","s":197.251,"e":197.331},{"text":"data,","s":197.411,"e":197.971},{"text":"from","s":198.051,"e":198.211},{"text":"real","s":198.291,"e":198.531},{"text":"inputs,","s":198.611,"e":199.251},{"text":"with","s":199.331,"e":199.411},{"text":"experts.","s":199.491,"e":200.371},{"text":"Thirty","s":200.611,"e":200.851},{"text":"labels","s":200.931,"e":201.331},{"text":"find","s":201.411,"e":201.651},{"text":"failures,","s":201.731,"e":202.611},{"text":"hundreds","s":202.651,"e":203.171},{"text":"measure","s":203.251,"e":203.491},{"text":"them.","s":203.571,"e":204.051},{"text":"Look","s":204.291,"e":204.531},{"text":"per","s":204.611,"e":204.771},{"text":"question.","s":204.851,"e":205.611},{"text":"Test","s":205.811,"e":206.131},{"text":"every","s":206.291,"e":206.531},{"text":"change,","s":206.691,"e":207.251},{"text":"even","s":207.371,"e":207.491},{"text":"wording.","s":207.571,"e":208.371},{"text":"And","s":208.451,"e":208.531},{"text":"pin","s":208.691,"e":208.931},{"text":"the","s":208.971,"e":209.011},{"text":"version,","s":209.091,"e":209.731},{"text":"because","s":209.851,"e":209.971},{"text":"a","s":210.011,"e":210.051},{"text":"better","s":210.131,"e":210.291},{"text":"model","s":210.371,"e":210.691},{"text":"can","s":210.771,"e":211.011},{"text":"still","s":211.091,"e":211.331},{"text":"move","s":211.411,"e":211.571},{"text":"your","s":211.651,"e":211.691},{"text":"thresholds.","s":211.731,"e":213.011},{"text":"Next:","s":213.571,"e":214.011},{"text":"deciding","s":214.131,"e":214.451},{"text":"responsibly.","s":214.931,"e":215.651}]}],"total":217.615,"audio":"/audio/20-evaluate.mp3"}