{"id":"18-judgments-as-data","number":18,"slug":"judgments-as-data","title":"Judgments as data","short":"Judgments as data","module":"Real systems","summary":"At a fraction of a cent per answer, judgments stop being calls you make and become data you keep: columns you can sort, filter, weight and train on. Here is a study that coded half a million police narratives for $25, a loop that turned tasting notes into features for a classic model, and why cheaper judgment will mean much more of it.","voice":"Jessica","scenes":[{"template":"title","chapter":"Intro","narration":"When a judgment costs a fraction of a cent, you stop thinking of it as a call. It becomes data: a column you compute once, store, and use in a hundred ways.","params":{"kicker":"Lesson 18","module":"Module 5 · Real systems","title":"Judgments as *data*","subtitle":"Compute once. Sort, filter, weight and train forever.","cast":["snap","curly"]},"start":0.5,"dur":11.727,"words":[{"text":"When","s":0.8,"e":0.96},{"text":"a","s":1,"e":1.04},{"text":"judgment","s":1.12,"e":1.52},{"text":"costs","s":1.6,"e":1.92},{"text":"a","s":2,"e":2.08},{"text":"fraction","s":2.16,"e":2.56},{"text":"of","s":2.6,"e":2.64},{"text":"a","s":2.72,"e":2.8},{"text":"cent,","s":3,"e":3.44},{"text":"you","s":3.52,"e":3.6},{"text":"stop","s":3.72,"e":3.84},{"text":"thinking","s":3.92,"e":4.24},{"text":"of","s":4.28,"e":4.32},{"text":"it","s":4.36,"e":4.4},{"text":"as","s":4.48,"e":4.56},{"text":"a","s":4.6,"e":4.64},{"text":"call.","s":4.72,"e":5.52},{"text":"It","s":5.56,"e":5.6},{"text":"becomes","s":5.68,"e":6},{"text":"data:","s":6.08,"e":6.96},{"text":"a","s":7,"e":7.04},{"text":"column","s":7.12,"e":7.44},{"text":"you","s":7.48,"e":7.52},{"text":"compute","s":7.6,"e":7.92},{"text":"once,","s":8,"e":8.64},{"text":"store,","s":8.88,"e":9.28},{"text":"and","s":9.36,"e":9.44},{"text":"use","s":9.6,"e":9.84},{"text":"in","s":9.88,"e":9.92},{"text":"a","s":10,"e":10.16},{"text":"hundred","s":10.2,"e":10.48},{"text":"ways.","s":10.56,"e":11.36}]},{"template":"code","chapter":"Ask once, reuse","narration":"The habit is simple. Store the raw answers: every probability, with the model version and the date. Then your weights, thresholds and filters are just code over a table. Change a weight and the ranking updates instantly, with no new model calls. One tester put it neatly: the judgments are cached, so rewriting the composition rules costs nothing.","params":{"file":"judgments.sql","lang":"text","lines":["-- one row per item, one column per question (raw probabilities)","CREATE TABLE ticket_judgments (","  ticket_id TEXT, model TEXT, judged_at DATE,","  refund_requested REAL, duplicate_charge REAL,","  churn_threat REAL, frustration REAL);","","-- change the weights any time: no new model calls","SELECT ticket_id,","  0.5*churn_threat + 0.3*frustration/2 + 0.2*duplicate_charge AS priority","FROM ticket_judgments ORDER BY priority DESC;"],"hl":[{"from":3,"to":5,"label":"raw answers + model + date","tone":"agent","at":"Store the raw answers"},{"from":8,"to":10,"label":"composites are just code","tone":"good","at":"Change a weight"}]},"start":12.227,"dur":23.389,"words":[{"text":"The","s":12.527,"e":12.687},{"text":"habit","s":12.847,"e":13.007},{"text":"is","s":13.087,"e":13.167},{"text":"simple.","s":13.247,"e":13.727},{"text":"Store","s":14.127,"e":14.447},{"text":"the","s":14.487,"e":14.527},{"text":"raw","s":14.687,"e":14.927},{"text":"answers:","s":15.007,"e":15.887},{"text":"every","s":16.007,"e":16.127},{"text":"probability,","s":16.207,"e":17.167},{"text":"with","s":17.247,"e":17.327},{"text":"the","s":17.367,"e":17.407},{"text":"model","s":17.487,"e":17.727},{"text":"version","s":17.807,"e":18.207},{"text":"and","s":18.367,"e":18.527},{"text":"the","s":18.567,"e":18.607},{"text":"date.","s":18.687,"e":19.087},{"text":"Then","s":19.487,"e":19.727},{"text":"your","s":19.887,"e":20.047},{"text":"weights,","s":20.127,"e":20.687},{"text":"thresholds","s":20.967,"e":21.287},{"text":"and","s":21.327,"e":21.407},{"text":"filters","s":21.487,"e":21.967},{"text":"are","s":22.047,"e":22.127},{"text":"just","s":22.207,"e":22.367},{"text":"code","s":22.447,"e":22.767},{"text":"over","s":22.887,"e":23.007},{"text":"a","s":23.047,"e":23.087},{"text":"table.","s":23.167,"e":23.567},{"text":"Change","s":24.047,"e":24.287},{"text":"a","s":24.327,"e":24.367},{"text":"weight","s":24.527,"e":24.687},{"text":"and","s":24.767,"e":24.847},{"text":"the","s":24.887,"e":24.927},{"text":"ranking","s":25.007,"e":25.327},{"text":"updates","s":25.407,"e":25.807},{"text":"instantly,","s":25.887,"e":26.527},{"text":"with","s":26.647,"e":26.767},{"text":"no","s":26.927,"e":27.087},{"text":"new","s":27.167,"e":27.247},{"text":"model","s":27.327,"e":27.567},{"text":"calls.","s":27.647,"e":28.367},{"text":"One","s":28.607,"e":28.767},{"text":"tester","s":28.967,"e":29.167},{"text":"put","s":29.247,"e":29.327},{"text":"it","s":29.367,"e":29.407},{"text":"neatly:","s":29.487,"e":30.367},{"text":"the","s":30.407,"e":30.447},{"text":"judgments","s":30.527,"e":30.927},{"text":"are","s":31.007,"e":31.167},{"text":"cached,","s":31.407,"e":31.887},{"text":"so","s":32.007,"e":32.127},{"text":"rewriting","s":32.207,"e":32.607},{"text":"the","s":32.687,"e":32.727},{"text":"composition","s":32.767,"e":33.327},{"text":"rules","s":33.407,"e":33.727},{"text":"costs","s":33.807,"e":34.207},{"text":"nothing.","s":34.287,"e":35.007}]},{"template":"stat","chapter":"Whole corpora","narration":"That changes what's affordable. Texas has five million police crash narratives that analysts rarely read, because coding them with a frontier model is too expensive. A research team screened half a million of them with ten questions, for twenty five dollars. Then they coded almost two hundred thousand with twenty seven questions, for thirty dollars, with zero failed calls. The whole five million would cost about seven hundred seventy four.","params":{"title":"Police crash narratives, Texas, with Jev","items":[{"value":499500,"label":"screened, 10 questions each","tone":"agent","at":"half a million"},{"value":25,"prefix":"$","label":"for the screening run","tone":"good","at":"twenty five dollars"},{"value":30,"prefix":"$","label":"to code 195,857 with 27 questions","tone":"good","at":"thirty dollars","hero":true}],"src":"Rafe & Das (Texas State University), Calibrated Decisions at Scale, arXiv 2026. Projected for all 5,018,080 narratives: $774.","punch":false},"start":35.616,"dur":28.274,"words":[{"text":"That","s":35.916,"e":36.156},{"text":"changes","s":36.236,"e":36.636},{"text":"what's","s":36.716,"e":36.956},{"text":"affordable.","s":37.036,"e":37.676},{"text":"Texas","s":38.156,"e":38.556},{"text":"has","s":38.676,"e":38.796},{"text":"five","s":38.876,"e":39.196},{"text":"million","s":39.436,"e":39.836},{"text":"police","s":39.916,"e":40.236},{"text":"crash","s":40.316,"e":40.556},{"text":"narratives","s":40.636,"e":41.036},{"text":"that","s":41.156,"e":41.276},{"text":"analysts","s":41.396,"e":41.836},{"text":"rarely","s":41.996,"e":42.316},{"text":"read,","s":42.476,"e":43.116},{"text":"because","s":43.236,"e":43.356},{"text":"coding","s":43.436,"e":43.756},{"text":"them","s":43.836,"e":43.916},{"text":"with","s":43.956,"e":43.996},{"text":"a","s":44.036,"e":44.076},{"text":"frontier","s":44.156,"e":44.556},{"text":"model","s":44.636,"e":44.956},{"text":"is","s":45.036,"e":45.276},{"text":"too","s":45.356,"e":45.436},{"text":"expensive.","s":45.516,"e":46.316},{"text":"A","s":46.716,"e":46.796},{"text":"research","s":46.876,"e":47.196},{"text":"team","s":47.276,"e":47.516},{"text":"screened","s":47.596,"e":47.996},{"text":"half","s":48.116,"e":48.236},{"text":"a","s":48.276,"e":48.316},{"text":"million","s":48.396,"e":48.636},{"text":"of","s":48.716,"e":48.796},{"text":"them","s":48.916,"e":49.036},{"text":"with","s":49.116,"e":49.276},{"text":"ten","s":49.356,"e":49.516},{"text":"questions,","s":49.596,"e":50.236},{"text":"for","s":50.316,"e":50.396},{"text":"twenty","s":50.476,"e":50.636},{"text":"five","s":50.716,"e":50.876},{"text":"dollars.","s":50.956,"e":51.916},{"text":"Then","s":51.996,"e":52.236},{"text":"they","s":52.396,"e":52.476},{"text":"coded","s":52.556,"e":52.796},{"text":"almost","s":52.876,"e":53.196},{"text":"two","s":53.276,"e":53.436},{"text":"hundred","s":53.516,"e":53.756},{"text":"thousand","s":53.996,"e":54.476},{"text":"with","s":54.596,"e":54.716},{"text":"twenty","s":54.836,"e":54.956},{"text":"seven","s":55.116,"e":55.356},{"text":"questions,","s":55.436,"e":56.156},{"text":"for","s":56.276,"e":56.396},{"text":"thirty","s":56.436,"e":56.636},{"text":"dollars,","s":56.716,"e":57.436},{"text":"with","s":57.516,"e":57.596},{"text":"zero","s":57.836,"e":58.156},{"text":"failed","s":58.316,"e":58.556},{"text":"calls.","s":58.636,"e":59.516},{"text":"The","s":59.596,"e":59.676},{"text":"whole","s":59.836,"e":59.996},{"text":"five","s":60.076,"e":60.316},{"text":"million","s":60.476,"e":60.876},{"text":"would","s":60.956,"e":61.116},{"text":"cost","s":61.236,"e":61.356},{"text":"about","s":61.516,"e":61.676},{"text":"seven","s":61.756,"e":61.916},{"text":"hundred","s":61.996,"e":62.156},{"text":"seventy","s":62.316,"e":62.636},{"text":"four.","s":62.716,"e":63.276}]},{"template":"flow","chapter":"Whole corpora","narration":"Notice the two stages. A lean screen with ten presence questions runs over everything. The detailed twenty seven question schema only runs where the screen found something to code. The raw probabilities overstated how common things were, but once recalibrated on human labels, the review budget became a number: how many records a person would still need to read.","params":{"title":"Screen everything, code what matters, review what's uncertain","nodes":[{"label":"Screen","sub":"10 presence Nouls on every record","kind":"snap","edge":"a hit","at":"lean screen"},{"label":"Code","sub":"27 questions where it applies","kind":"snap","edge":"uncertain","at":"detailed"},{"label":"Review budget","sub":"records a person must read","kind":"you","at":"review"}]},"start":63.89,"dur":23.389,"words":[{"text":"Notice","s":64.19,"e":64.557},{"text":"the","s":64.59,"e":64.723},{"text":"two","s":64.75,"e":64.883},{"text":"stages.","s":64.91,"e":65.71},{"text":"A","s":66.11,"e":66.19},{"text":"lean","s":66.27,"e":66.51},{"text":"screen","s":66.59,"e":66.91},{"text":"with","s":67.03,"e":67.15},{"text":"ten","s":67.23,"e":67.47},{"text":"presence","s":67.55,"e":67.87},{"text":"questions","s":67.95,"e":68.43},{"text":"runs","s":68.51,"e":68.75},{"text":"over","s":68.91,"e":69.07},{"text":"everything.","s":69.15,"e":70.03},{"text":"The","s":70.27,"e":70.43},{"text":"detailed","s":70.47,"e":70.91},{"text":"twenty","s":70.99,"e":71.23},{"text":"seven","s":71.283,"e":71.55},{"text":"question","s":71.63,"e":71.95},{"text":"schema","s":72.03,"e":72.51},{"text":"only","s":72.75,"e":72.99},{"text":"runs","s":73.07,"e":73.31},{"text":"where","s":73.39,"e":73.47},{"text":"the","s":73.51,"e":73.55},{"text":"screen","s":73.63,"e":73.95},{"text":"found","s":74.07,"e":74.19},{"text":"something","s":74.35,"e":74.51},{"text":"to","s":74.59,"e":74.67},{"text":"code.","s":74.87,"e":75.63},{"text":"The","s":75.67,"e":75.71},{"text":"raw","s":75.79,"e":75.95},{"text":"probabilities","s":76.03,"e":76.75},{"text":"overstated","s":76.83,"e":77.55},{"text":"how","s":77.63,"e":77.71},{"text":"common","s":77.79,"e":78.03},{"text":"things","s":78.11,"e":78.35},{"text":"were,","s":78.51,"e":79.07},{"text":"but","s":79.23,"e":79.39},{"text":"once","s":79.47,"e":79.63},{"text":"recalibrated","s":79.71,"e":80.43},{"text":"on","s":80.51,"e":80.59},{"text":"human","s":80.71,"e":80.83},{"text":"labels,","s":80.91,"e":81.71},{"text":"the","s":81.75,"e":81.79},{"text":"review","s":81.95,"e":82.11},{"text":"budget","s":82.19,"e":82.51},{"text":"became","s":82.59,"e":82.83},{"text":"a","s":82.87,"e":82.91},{"text":"number:","s":82.99,"e":83.79},{"text":"how","s":83.87,"e":83.95},{"text":"many","s":84.03,"e":84.19},{"text":"records","s":84.27,"e":84.59},{"text":"a","s":84.67,"e":84.75},{"text":"person","s":84.91,"e":85.07},{"text":"would","s":85.15,"e":85.39},{"text":"still","s":85.47,"e":85.63},{"text":"need","s":85.71,"e":85.87},{"text":"to","s":85.95,"e":86.03},{"text":"read.","s":86.11,"e":86.67}]},{"template":"table","chapter":"Judgments as features","narration":"Judgments also make great features for classic machine learning. A TypeSafe cookbook predicts a wine critic's score from the tasting note. Predicting the average gives an error of three point oh nine points. Word counts get two point four seven. Asking the decision model for the score directly, two point one five. But thirty eight narrow questions, fed as features to a CatBoost model, got it down to one point seven seven.","params":{"title":"Predict a critic's score (80–100) from a tasting note","cols":["Approach","Held-out error (RMSE, points)"],"widths":"2fr 1fr","rows":[{"cells":["Always predict the average","3.09"],"at":"three point oh nine"},{"cells":["CatBoost on word counts","2.47"],"at":"Word counts"},{"cells":["Ask the decision model for the score","2.15"],"at":"directly"},{"cells":["*38 narrow questions → CatBoost*","*1.77*"],"hl":true,"at":"thirty eight narrow questions"}],"note":"TypeSafe cookbook: Autoresearch feature discovery, 2,000 wine reviews, 800 held out."},"start":87.279,"dur":26.759,"words":[{"text":"Judgments","s":87.579,"e":88.139},{"text":"also","s":88.219,"e":88.539},{"text":"make","s":88.619,"e":88.779},{"text":"great","s":88.859,"e":89.179},{"text":"features","s":89.259,"e":89.659},{"text":"for","s":89.739,"e":89.899},{"text":"classic","s":89.979,"e":90.379},{"text":"machine","s":90.459,"e":90.699},{"text":"learning.","s":90.779,"e":91.459},{"text":"A","s":91.739,"e":91.819},{"text":"TypeSafe","s":91.939,"e":92.299},{"text":"cookbook","s":92.379,"e":92.859},{"text":"predicts","s":93.019,"e":93.219},{"text":"a","s":93.259,"e":93.339},{"text":"wine","s":93.419,"e":93.579},{"text":"critic's","s":93.659,"e":93.979},{"text":"score","s":94.059,"e":94.379},{"text":"from","s":94.459,"e":94.539},{"text":"the","s":94.579,"e":94.619},{"text":"tasting","s":94.699,"e":95.099},{"text":"note.","s":95.259,"e":95.619},{"text":"Predicting","s":95.819,"e":96.139},{"text":"the","s":96.219,"e":96.299},{"text":"average","s":96.459,"e":96.939},{"text":"gives","s":96.979,"e":97.099},{"text":"an","s":97.179,"e":97.259},{"text":"error","s":97.339,"e":97.659},{"text":"of","s":97.739,"e":97.819},{"text":"three","s":97.899,"e":98.059},{"text":"point","s":98.219,"e":98.299},{"text":"oh","s":98.539,"e":98.619},{"text":"nine","s":98.699,"e":98.859},{"text":"points.","s":99.019,"e":99.899},{"text":"Word","s":99.979,"e":100.139},{"text":"counts","s":100.219,"e":100.539},{"text":"get","s":100.619,"e":100.859},{"text":"two","s":100.939,"e":101.019},{"text":"point","s":101.099,"e":101.339},{"text":"four","s":101.459,"e":101.579},{"text":"seven.","s":101.779,"e":102.299},{"text":"Asking","s":102.379,"e":102.699},{"text":"the","s":102.779,"e":102.819},{"text":"decision","s":102.859,"e":103.259},{"text":"model","s":103.339,"e":103.499},{"text":"for","s":103.579,"e":103.659},{"text":"the","s":103.699,"e":103.739},{"text":"score","s":103.819,"e":104.059},{"text":"directly,","s":104.259,"e":105.019},{"text":"two","s":105.099,"e":105.179},{"text":"point","s":105.259,"e":105.499},{"text":"one","s":105.619,"e":105.739},{"text":"five.","s":105.939,"e":106.619},{"text":"But","s":106.739,"e":106.859},{"text":"thirty","s":106.939,"e":107.259},{"text":"eight","s":107.339,"e":107.499},{"text":"narrow","s":107.659,"e":107.979},{"text":"questions,","s":108.059,"e":108.859},{"text":"fed","s":108.939,"e":109.019},{"text":"as","s":109.099,"e":109.179},{"text":"features","s":109.259,"e":109.659},{"text":"to","s":109.699,"e":109.739},{"text":"a","s":109.819,"e":109.899},{"text":"CatBoost","s":110.019,"e":110.379},{"text":"model,","s":110.459,"e":110.939},{"text":"got","s":111.019,"e":111.099},{"text":"it","s":111.179,"e":111.259},{"text":"down","s":111.339,"e":111.499},{"text":"to","s":111.579,"e":111.739},{"text":"one","s":111.819,"e":111.979},{"text":"point","s":112.059,"e":112.379},{"text":"seven","s":112.539,"e":112.699},{"text":"seven.","s":112.779,"e":113.339}]},{"template":"flow","chapter":"Let the data pick the questions","narration":"And nobody wrote those thirty eight questions by hand. An LLM proposed them, the decision model answered them for every row, and CatBoost reported which questions helped and which rows it still got wrong. The next round read that report and proposed better questions. Five rounds took the error from one point eight seven to one point seven seven. Trying a change costs no API calls, because the answers are already stored.","params":{"title":"An autoresearch loop","nodes":[{"label":"LLM proposes questions","sub":"reads last round's errors","kind":"mull","edge":"questions","at":"An LLM proposed them"},{"label":"Decider answers every row","sub":"probabilities become columns","kind":"snap","edge":"features","at":"answered them"},{"label":"CatBoost trains","sub":"reports what helped","kind":"curly","at":"CatBoost reported"}],"caption":"Keep a question only if held-out error improves.","caption_at":"five rounds"},"start":114.038,"dur":27.072,"words":[{"text":"And","s":114.338,"e":114.578},{"text":"nobody","s":114.658,"e":115.058},{"text":"wrote","s":115.138,"e":115.378},{"text":"those","s":115.458,"e":115.618},{"text":"thirty","s":115.698,"e":115.938},{"text":"eight","s":116.018,"e":116.098},{"text":"questions","s":116.178,"e":116.578},{"text":"by","s":116.658,"e":116.738},{"text":"hand.","s":116.818,"e":117.378},{"text":"An","s":117.778,"e":117.938},{"text":"LLM","s":118.018,"e":118.498},{"text":"proposed","s":118.578,"e":119.058},{"text":"them,","s":119.178,"e":119.618},{"text":"the","s":119.658,"e":119.698},{"text":"decision","s":119.898,"e":120.098},{"text":"model","s":120.178,"e":120.418},{"text":"answered","s":120.498,"e":120.898},{"text":"them","s":120.978,"e":121.058},{"text":"for","s":121.138,"e":121.218},{"text":"every","s":121.298,"e":121.458},{"text":"row,","s":121.618,"e":122.258},{"text":"and","s":122.338,"e":122.418},{"text":"CatBoost","s":122.498,"e":122.978},{"text":"reported","s":123.058,"e":123.458},{"text":"which","s":123.538,"e":123.698},{"text":"questions","s":123.778,"e":124.178},{"text":"helped","s":124.258,"e":124.578},{"text":"and","s":124.658,"e":124.738},{"text":"which","s":124.818,"e":125.058},{"text":"rows","s":125.178,"e":125.298},{"text":"it","s":125.378,"e":125.458},{"text":"still","s":125.538,"e":125.698},{"text":"got","s":125.778,"e":125.938},{"text":"wrong.","s":126.018,"e":126.418},{"text":"The","s":126.898,"e":126.978},{"text":"next","s":127.058,"e":127.298},{"text":"round","s":127.458,"e":127.618},{"text":"read","s":127.698,"e":127.858},{"text":"that","s":127.938,"e":128.018},{"text":"report","s":128.258,"e":128.498},{"text":"and","s":128.538,"e":128.578},{"text":"proposed","s":128.658,"e":129.058},{"text":"better","s":129.218,"e":129.378},{"text":"questions.","s":129.458,"e":130.578},{"text":"Five","s":130.658,"e":130.898},{"text":"rounds","s":130.978,"e":131.378},{"text":"took","s":131.458,"e":131.538},{"text":"the","s":131.618,"e":131.698},{"text":"error","s":131.778,"e":132.018},{"text":"from","s":132.098,"e":132.258},{"text":"one","s":132.338,"e":132.418},{"text":"point","s":132.578,"e":132.658},{"text":"eight","s":132.818,"e":132.978},{"text":"seven","s":133.058,"e":133.298},{"text":"to","s":133.538,"e":133.778},{"text":"one","s":133.858,"e":133.938},{"text":"point","s":134.098,"e":134.418},{"text":"seven","s":134.538,"e":134.658},{"text":"seven.","s":134.818,"e":135.698},{"text":"Trying","s":135.858,"e":136.018},{"text":"a","s":136.058,"e":136.098},{"text":"change","s":136.178,"e":136.498},{"text":"costs","s":136.578,"e":136.978},{"text":"no","s":137.138,"e":137.298},{"text":"API","s":137.378,"e":137.778},{"text":"calls,","s":137.858,"e":138.338},{"text":"because","s":138.458,"e":138.578},{"text":"the","s":138.658,"e":138.738},{"text":"answers","s":138.938,"e":139.138},{"text":"are","s":139.218,"e":139.298},{"text":"already","s":139.378,"e":139.618},{"text":"stored.","s":139.698,"e":140.498}]},{"template":"statement","chapter":"Jevons","narration":"Which brings us to the name. TypeSafe named Jev after William Stanley Jevons, who noticed in 1865 that more efficient steam engines made Britain burn more coal, not less. Their bet: every order of magnitude drop in the cost of intelligence unlocks orders of magnitude more uses. Whether that holds depends on demand, but it's the right way to think about judgments as data.","params":{"eyebrow":"Why it's called Jev","text":"Cheaper judgment means *more* judgment, not less.","sub":"Jevons (1865): efficient engines raised coal use. TypeSafe's bet: cheaper intelligence unlocks far more uses.","sub_at":"more uses","art":"coin","tone":"agent","punch":"more coal"},"start":141.11,"dur":25.949,"words":[{"text":"Which","s":141.41,"e":141.65},{"text":"brings","s":141.73,"e":141.89},{"text":"us","s":141.97,"e":142.13},{"text":"to","s":142.21,"e":142.29},{"text":"the","s":142.33,"e":142.37},{"text":"name.","s":142.45,"e":143.09},{"text":"TypeSafe","s":143.33,"e":143.89},{"text":"named","s":144.05,"e":144.21},{"text":"Jev","s":144.29,"e":144.69},{"text":"after","s":144.77,"e":145.01},{"text":"William","s":145.17,"e":145.49},{"text":"Stanley","s":145.57,"e":145.97},{"text":"Jevons,","s":146.13,"e":146.77},{"text":"who","s":146.85,"e":146.93},{"text":"noticed","s":147.01,"e":147.33},{"text":"in","s":147.41,"e":147.49},{"text":"1865","s":147.57,"e":148.69},{"text":"that","s":148.77,"e":148.93},{"text":"more","s":149.01,"e":149.17},{"text":"efficient","s":149.25,"e":149.65},{"text":"steam","s":149.81,"e":150.05},{"text":"engines","s":150.13,"e":150.53},{"text":"made","s":150.61,"e":150.77},{"text":"Britain","s":150.85,"e":151.17},{"text":"burn","s":151.25,"e":151.49},{"text":"more","s":151.65,"e":151.81},{"text":"coal,","s":151.89,"e":152.45},{"text":"not","s":152.57,"e":152.69},{"text":"less.","s":152.93,"e":153.41},{"text":"Their","s":153.65,"e":153.81},{"text":"bet:","s":153.89,"e":154.61},{"text":"every","s":154.69,"e":154.93},{"text":"order","s":155.09,"e":155.25},{"text":"of","s":155.33,"e":155.41},{"text":"magnitude","s":155.49,"e":155.89},{"text":"drop","s":155.97,"e":156.29},{"text":"in","s":156.33,"e":156.37},{"text":"the","s":156.41,"e":156.45},{"text":"cost","s":156.53,"e":156.77},{"text":"of","s":156.85,"e":156.89},{"text":"intelligence","s":156.93,"e":157.65},{"text":"unlocks","s":157.81,"e":158.21},{"text":"orders","s":158.37,"e":158.69},{"text":"of","s":158.77,"e":158.85},{"text":"magnitude","s":158.93,"e":159.41},{"text":"more","s":159.57,"e":159.81},{"text":"uses.","s":159.89,"e":160.69},{"text":"Whether","s":160.93,"e":161.17},{"text":"that","s":161.25,"e":161.33},{"text":"holds","s":161.41,"e":161.81},{"text":"depends","s":161.89,"e":162.21},{"text":"on","s":162.29,"e":162.37},{"text":"demand,","s":162.65,"e":163.17},{"text":"but","s":163.29,"e":163.41},{"text":"it's","s":163.49,"e":163.65},{"text":"the","s":163.73,"e":163.81},{"text":"right","s":163.89,"e":164.05},{"text":"way","s":164.13,"e":164.21},{"text":"to","s":164.29,"e":164.37},{"text":"think","s":164.45,"e":164.61},{"text":"about","s":164.73,"e":164.85},{"text":"judgments","s":165.01,"e":165.49},{"text":"as","s":165.57,"e":165.73},{"text":"data.","s":165.81,"e":166.45}]},{"template":"quiz","chapter":"Quick check","narration":"Quick check. You scored ten thousand tickets with six questions last week. Today the team wants to change the priority weights. What do you need to do?","params":{"q":"10,000 tickets were scored with 6 questions last week. New priority **weights** today. What now?","options":["Rescore all 10,000 tickets","Recompute the priority in code from the stored answers","Ask one new question per ticket","Retrain the decision model"],"answer":1,"explain":"The judgments didn't change, only how you combine them. If you stored the raw answers, a new weighting is a query, not a model call."},"start":167.059,"dur":14.486,"words":[{"text":"Quick","s":167.359,"e":167.599},{"text":"check.","s":167.759,"e":168.239},{"text":"You","s":168.319,"e":168.399},{"text":"scored","s":168.479,"e":168.799},{"text":"ten","s":168.879,"e":169.039},{"text":"thousand","s":169.279,"e":169.679},{"text":"tickets","s":169.759,"e":170.159},{"text":"with","s":170.279,"e":170.399},{"text":"six","s":170.479,"e":170.719},{"text":"questions","s":170.799,"e":171.199},{"text":"last","s":171.279,"e":171.439},{"text":"week.","s":171.519,"e":172.239},{"text":"Today","s":172.319,"e":172.639},{"text":"the","s":172.719,"e":172.799},{"text":"team","s":172.919,"e":173.039},{"text":"wants","s":173.119,"e":173.279},{"text":"to","s":173.359,"e":173.439},{"text":"change","s":173.599,"e":173.759},{"text":"the","s":173.799,"e":173.839},{"text":"priority","s":173.919,"e":174.399},{"text":"weights.","s":174.639,"e":175.199},{"text":"What","s":175.279,"e":175.359},{"text":"do","s":175.399,"e":175.439},{"text":"you","s":175.479,"e":175.519},{"text":"need","s":175.599,"e":175.679},{"text":"to","s":175.719,"e":175.759},{"text":"do?","s":175.839,"e":176.399}]},{"template":"exercise","chapter":"Try it","narration":"Try it. Have your agent turn one judgment in your product into a stored, reusable table.","params":{"title":"Make judgments a table","intro":"Paste this into your coding agent:","prompt":"Pick one judgment we make about records (tickets, leads, reviews,\ndocuments). Create a table that stores the raw answer for each question\n(probabilities), the model ID and the date, keyed by record. Backfill\n1,000 records in batches and report the measured cost per 1,000. Move\nthe weighting and thresholds into a query or function over that table,\nand show that changing a weight needs no new model calls."},"start":181.545,"dur":6.828,"words":[{"text":"Try","s":181.845,"e":182.085},{"text":"it.","s":182.165,"e":182.645},{"text":"Have","s":182.725,"e":182.885},{"text":"your","s":182.965,"e":183.045},{"text":"agent","s":183.205,"e":183.365},{"text":"turn","s":183.445,"e":183.685},{"text":"one","s":183.845,"e":184.005},{"text":"judgment","s":184.085,"e":184.485},{"text":"in","s":184.525,"e":184.565},{"text":"your","s":184.645,"e":184.725},{"text":"product","s":184.805,"e":185.205},{"text":"into","s":185.285,"e":185.525},{"text":"a","s":185.605,"e":185.685},{"text":"stored,","s":185.925,"e":186.325},{"text":"reusable","s":186.445,"e":186.965},{"text":"table.","s":187.045,"e":187.765}]},{"template":"recap","chapter":"Recap","narration":"So: judgments are data. Store the raw answers with the model and date, and compute composites in code. Whole corpora are affordable now, so screen first, then code. Use judgments as features, and let the data pick the questions. That's module five. Next, module six: shipping it, starting with the jagged edges.","params":{"items":["Store raw answers + model + date; composites are code.","Corpora are affordable: 499,500 narratives screened for $25.","Judgments make features: 38 questions beat every baseline.","Cheaper judgment means more judgment (Jevons)."],"next":"Know the jagged edges","next_at":"Next"},"start":188.373,"dur":23.869,"words":[{"text":"So:","s":188.673,"e":189.473},{"text":"judgments","s":189.553,"e":190.033},{"text":"are","s":190.113,"e":190.273},{"text":"data.","s":190.353,"e":190.833},{"text":"Store","s":191.313,"e":191.553},{"text":"the","s":191.633,"e":191.673},{"text":"raw","s":191.713,"e":191.953},{"text":"answers","s":192.033,"e":192.433},{"text":"with","s":192.513,"e":192.593},{"text":"the","s":192.633,"e":192.673},{"text":"model","s":192.753,"e":193.073},{"text":"and","s":193.113,"e":193.153},{"text":"date,","s":193.233,"e":193.713},{"text":"and","s":193.753,"e":193.793},{"text":"compute","s":193.873,"e":194.193},{"text":"composites","s":194.273,"e":194.833},{"text":"in","s":194.913,"e":194.993},{"text":"code.","s":195.073,"e":195.873},{"text":"Whole","s":195.953,"e":196.113},{"text":"corpora","s":196.193,"e":196.673},{"text":"are","s":196.753,"e":196.833},{"text":"affordable","s":196.953,"e":197.313},{"text":"now,","s":197.393,"e":197.793},{"text":"so","s":197.873,"e":198.033},{"text":"screen","s":198.193,"e":198.513},{"text":"first,","s":198.593,"e":199.153},{"text":"then","s":199.233,"e":199.393},{"text":"code.","s":199.473,"e":200.353},{"text":"Use","s":200.433,"e":200.593},{"text":"judgments","s":200.673,"e":201.073},{"text":"as","s":201.153,"e":201.233},{"text":"features,","s":201.393,"e":201.953},{"text":"and","s":201.993,"e":202.033},{"text":"let","s":202.113,"e":202.193},{"text":"the","s":202.273,"e":202.353},{"text":"data","s":202.433,"e":202.673},{"text":"pick","s":202.753,"e":202.913},{"text":"the","s":202.953,"e":202.993},{"text":"questions.","s":203.073,"e":204.113},{"text":"That's","s":204.193,"e":204.433},{"text":"module","s":204.513,"e":204.753},{"text":"five.","s":204.913,"e":205.793},{"text":"Next,","s":205.873,"e":206.513},{"text":"module","s":206.593,"e":206.913},{"text":"six:","s":207.073,"e":207.873},{"text":"shipping","s":207.953,"e":208.273},{"text":"it,","s":208.353,"e":208.753},{"text":"starting","s":208.833,"e":209.153},{"text":"with","s":209.233,"e":209.313},{"text":"the","s":209.353,"e":209.393},{"text":"jagged","s":209.553,"e":209.873},{"text":"edges.","s":209.953,"e":210.673}]}],"total":212.642,"audio":"/audio/18-judgments-as-data.mp3"}