{"id":"06-choice","number":6,"slug":"choice-questions","title":"Choice: one answer from a set","short":"Choice","module":"Asking good questions","summary":"A Choice picks one option from a set you define and returns a probability for every option. Here is how to read the distribution and its confidence, why a none-of-the-above option matters, how contrastive descriptions sharpen close calls, and what reversing the option order did to 3,080 real answers.","voice":"Jessica","scenes":[{"template":"title","chapter":"Intro","narration":"Most decisions in software pick one thing from a list: which team, which category, which intent. That's a Choice. Let's see what it gives back, and the four habits that make it reliable.","params":{"kicker":"Lesson 6","module":"Module 2 · Asking good questions","title":"Choice: one answer *from a set*","subtitle":"A probability for every option you offer.","cast":["snap","curly"],"bubble":"returns · 0.63","bubble_at":"That's a Choice"},"start":0.5,"dur":13.086,"words":[{"text":"Most","s":0.8,"e":1.12},{"text":"decisions","s":1.2,"e":1.6},{"text":"in","s":1.68,"e":1.76},{"text":"software","s":1.84,"e":2.4},{"text":"pick","s":2.56,"e":2.8},{"text":"one","s":2.96,"e":3.12},{"text":"thing","s":3.2,"e":3.44},{"text":"from","s":3.52,"e":3.6},{"text":"a","s":3.68,"e":3.72},{"text":"list:","s":3.76,"e":4.48},{"text":"which","s":4.6,"e":4.72},{"text":"team,","s":4.8,"e":5.36},{"text":"which","s":5.44,"e":5.52},{"text":"category,","s":5.6,"e":6.4},{"text":"which","s":6.48,"e":6.56},{"text":"intent.","s":6.84,"e":7.2},{"text":"That's","s":7.6,"e":7.8},{"text":"a","s":7.84,"e":7.88},{"text":"Choice.","s":7.92,"e":8.6},{"text":"Let's","s":8.8,"e":9.04},{"text":"see","s":9.08,"e":9.12},{"text":"what","s":9.2,"e":9.28},{"text":"it","s":9.32,"e":9.36},{"text":"gives","s":9.4,"e":9.6},{"text":"back,","s":9.68,"e":10.16},{"text":"and","s":10.28,"e":10.4},{"text":"the","s":10.48,"e":10.56},{"text":"four","s":10.64,"e":10.8},{"text":"habits","s":10.88,"e":11.28},{"text":"that","s":11.36,"e":11.44},{"text":"make","s":11.52,"e":11.6},{"text":"it","s":11.68,"e":11.76},{"text":"reliable.","s":11.84,"e":12.72}]},{"template":"dist","chapter":"Reading a Choice","narration":"Here's a messy ticket. Shoes arrived late and in the wrong size, and there are two charges on the card. Which team should handle it? Returns wins at point six three, but billing gets point three three, because of the double charge. That split is honest. This ticket belongs to two teams.","params":{"state":"Shoes arrived two weeks late and in the wrong size. Also I see two charges of $120 on my card. What are you going to do about this?","question":"Which team should handle this?","options":[{"label":"returns","desc":"exchanges, wrong or damaged items","p":0.63},{"label":"billing","desc":"charges, invoices, payment problems","p":0.33},{"label":"shipping","desc":"delivery status, delays","p":0.04}],"at":"Returns wins","conf_at":"That split","formula":"confidence = (0.63 − 1/3) / (1 − 1/3) = 0.45"},"start":13.586,"dur":19.001,"words":[{"text":"Here's","s":13.886,"e":14.166},{"text":"a","s":14.206,"e":14.246},{"text":"messy","s":14.286,"e":14.606},{"text":"ticket.","s":14.686,"e":15.166},{"text":"Shoes","s":15.566,"e":15.806},{"text":"arrived","s":15.886,"e":16.126},{"text":"late","s":16.206,"e":16.526},{"text":"and","s":16.606,"e":16.846},{"text":"in","s":16.886,"e":16.926},{"text":"the","s":17.006,"e":17.046},{"text":"wrong","s":17.086,"e":17.326},{"text":"size,","s":17.406,"e":18.046},{"text":"and","s":18.166,"e":18.286},{"text":"there","s":18.366,"e":18.446},{"text":"are","s":18.526,"e":18.606},{"text":"two","s":18.686,"e":18.846},{"text":"charges","s":19.006,"e":19.326},{"text":"on","s":19.366,"e":19.406},{"text":"the","s":19.486,"e":19.526},{"text":"card.","s":19.566,"e":20.366},{"text":"Which","s":20.446,"e":20.526},{"text":"team","s":20.606,"e":20.766},{"text":"should","s":20.846,"e":20.926},{"text":"handle","s":21.006,"e":21.246},{"text":"it?","s":21.326,"e":21.566},{"text":"Returns","s":21.886,"e":22.366},{"text":"wins","s":22.526,"e":22.686},{"text":"at","s":22.766,"e":22.846},{"text":"point","s":22.926,"e":23.086},{"text":"six","s":23.246,"e":23.486},{"text":"three,","s":23.566,"e":24.126},{"text":"but","s":24.206,"e":24.286},{"text":"billing","s":24.446,"e":24.846},{"text":"gets","s":24.966,"e":25.086},{"text":"point","s":25.166,"e":25.406},{"text":"three","s":25.486,"e":25.646},{"text":"three,","s":25.806,"e":26.126},{"text":"because","s":26.206,"e":26.366},{"text":"of","s":26.446,"e":26.486},{"text":"the","s":26.526,"e":26.566},{"text":"double","s":26.606,"e":26.846},{"text":"charge.","s":26.926,"e":27.646},{"text":"That","s":27.886,"e":28.046},{"text":"split","s":28.126,"e":28.446},{"text":"is","s":28.526,"e":28.766},{"text":"honest.","s":28.846,"e":29.486},{"text":"This","s":29.806,"e":29.966},{"text":"ticket","s":30.086,"e":30.206},{"text":"belongs","s":30.286,"e":30.686},{"text":"to","s":30.726,"e":30.766},{"text":"two","s":30.926,"e":31.086},{"text":"teams.","s":31.166,"e":31.966}]},{"template":"statement","chapter":"Reading a Choice","narration":"Confidence turns that shape into one number. Zero means the probability is spread evenly. One means it's all on one option. Here it's point four five. It's computed from the probabilities, so you can always check it, or use your own measure, like the gap between the top two options.","params":{"eyebrow":"Confidence = how peaked the distribution is","text":"0 = spread evenly. 1 = all on one option. Here: *0.45*.","sub":"Computed from the probabilities: (p_max − 1/n) / (1 − 1/n). The top-two ratio works too.","sub_at":"computed from the probabilities","art":"gauge","tone":"agent","punch":"point four five"},"start":32.587,"dur":19.079,"words":[{"text":"Confidence","s":32.887,"e":33.447},{"text":"turns","s":33.527,"e":33.847},{"text":"that","s":33.967,"e":34.087},{"text":"shape","s":34.167,"e":34.487},{"text":"into","s":34.647,"e":34.807},{"text":"one","s":34.967,"e":35.127},{"text":"number.","s":35.207,"e":35.887},{"text":"Zero","s":36.167,"e":36.487},{"text":"means","s":36.567,"e":36.807},{"text":"the","s":36.887,"e":36.927},{"text":"probability","s":36.967,"e":37.607},{"text":"is","s":37.687,"e":37.767},{"text":"spread","s":37.927,"e":38.087},{"text":"evenly.","s":38.247,"e":39.127},{"text":"One","s":39.207,"e":39.447},{"text":"means","s":39.607,"e":39.767},{"text":"it's","s":39.847,"e":40.087},{"text":"all","s":40.167,"e":40.327},{"text":"on","s":40.407,"e":40.487},{"text":"one","s":40.567,"e":40.647},{"text":"option.","s":40.727,"e":41.527},{"text":"Here","s":41.687,"e":41.847},{"text":"it's","s":42.007,"e":42.247},{"text":"point","s":42.407,"e":42.567},{"text":"four","s":42.687,"e":42.807},{"text":"five.","s":42.887,"e":43.687},{"text":"It's","s":43.767,"e":43.887},{"text":"computed","s":43.927,"e":44.407},{"text":"from","s":44.487,"e":44.567},{"text":"the","s":44.607,"e":44.647},{"text":"probabilities,","s":44.727,"e":45.607},{"text":"so","s":45.727,"e":45.847},{"text":"you","s":45.927,"e":45.967},{"text":"can","s":46.007,"e":46.167},{"text":"always","s":46.247,"e":46.567},{"text":"check","s":46.647,"e":46.807},{"text":"it,","s":46.887,"e":47.287},{"text":"or","s":47.407,"e":47.527},{"text":"use","s":47.687,"e":47.767},{"text":"your","s":47.847,"e":47.927},{"text":"own","s":48.007,"e":48.087},{"text":"measure,","s":48.247,"e":48.647},{"text":"like","s":48.767,"e":48.887},{"text":"the","s":48.967,"e":49.047},{"text":"gap","s":49.167,"e":49.287},{"text":"between","s":49.447,"e":49.607},{"text":"the","s":49.647,"e":49.687},{"text":"top","s":49.767,"e":49.927},{"text":"two","s":50.007,"e":50.087},{"text":"options.","s":50.167,"e":50.967}]},{"template":"compare","chapter":"None of the above","narration":"Habit one: always offer a way out. Someone writes in to apply for a designer job. Your options are returns, shipping and billing. Without an other option, the model has to pick something: billing, at point five, spread thin. Add other, described as none of the above, and it goes there with all the probability.","params":{"title":"“I'd like to apply for the senior designer job you posted.”","left":{"title":"returns · shipping · billing","tone":"bad","items":[{"text":"billing **0.50** · shipping 0.29 · returns 0.21","at":"billing, at point five"},{"text":"Forced to pick something"}],"at":"Without an other option"},"right":{"title":"… + other: “None of the above”","tone":"good","items":[{"text":"other **1.00**","at":"all the probability"},{"text":"Your code can route it anywhere"}],"at":"Add other"},"vs":"vs"},"start":51.666,"dur":22.266,"words":[{"text":"Habit","s":51.966,"e":52.226},{"text":"one:","s":52.286,"e":52.686},{"text":"always","s":53.326,"e":53.726},{"text":"offer","s":53.806,"e":54.126},{"text":"a","s":54.206,"e":54.286},{"text":"way","s":54.366,"e":54.526},{"text":"out.","s":54.686,"e":55.326},{"text":"Someone","s":55.646,"e":55.966},{"text":"writes","s":56.086,"e":56.206},{"text":"in","s":56.286,"e":56.446},{"text":"to","s":56.486,"e":56.526},{"text":"apply","s":56.606,"e":56.926},{"text":"for","s":57.006,"e":57.086},{"text":"a","s":57.126,"e":57.166},{"text":"designer","s":57.366,"e":57.646},{"text":"job.","s":57.726,"e":58.046},{"text":"Your","s":58.366,"e":58.446},{"text":"options","s":58.526,"e":58.926},{"text":"are","s":59.006,"e":59.246},{"text":"returns,","s":59.406,"e":60.206},{"text":"shipping","s":60.286,"e":60.606},{"text":"and","s":60.686,"e":60.766},{"text":"billing.","s":60.846,"e":61.446},{"text":"Without","s":61.646,"e":61.886},{"text":"an","s":61.966,"e":62.046},{"text":"other","s":62.126,"e":62.366},{"text":"option,","s":62.446,"e":63.086},{"text":"the","s":63.126,"e":63.166},{"text":"model","s":63.246,"e":63.486},{"text":"has","s":63.646,"e":63.886},{"text":"to","s":63.926,"e":63.966},{"text":"pick","s":64.046,"e":64.206},{"text":"something:","s":64.446,"e":65.086},{"text":"billing,","s":65.166,"e":65.726},{"text":"at","s":65.806,"e":65.886},{"text":"point","s":65.966,"e":66.126},{"text":"five,","s":66.206,"e":66.846},{"text":"spread","s":66.926,"e":67.246},{"text":"thin.","s":67.326,"e":67.886},{"text":"Add","s":68.126,"e":68.286},{"text":"other,","s":68.446,"e":69.086},{"text":"described","s":69.166,"e":69.566},{"text":"as","s":69.646,"e":69.806},{"text":"none","s":69.966,"e":70.126},{"text":"of","s":70.166,"e":70.206},{"text":"the","s":70.246,"e":70.286},{"text":"above,","s":70.366,"e":70.926},{"text":"and","s":70.966,"e":71.006},{"text":"it","s":71.086,"e":71.126},{"text":"goes","s":71.166,"e":71.406},{"text":"there","s":71.526,"e":71.646},{"text":"with","s":71.806,"e":71.966},{"text":"all","s":72.046,"e":72.126},{"text":"the","s":72.206,"e":72.286},{"text":"probability.","s":72.366,"e":73.326}]},{"template":"request","chapter":"Describe the options","narration":"Habit two: describe the options. The model reads both the option names and their descriptions. For options that get confused, say what each one covers, what it's not for, and give an example or two. On a borderline question about where a disposable card works, plain names gave point nine. Contrastive descriptions gave a clean one point oh.","params":{"model":"jev-1.13.0","latency":0.25,"state":"Can I use a disposable card at any shop?","questions":[{"id":"plain","type":"choice","text":"Topic? (plain option names)","ans":{"choice":"card_limits","p":0.9},"say_at":"plain names"},{"id":"rich","type":"choice","text":"Topic? (what / not_for / examples per option)","ans":{"choice":"card_limits","p":1},"say_at":"Contrastive descriptions"}],"send_at":"On a borderline","ans_at":"plain names gave","hl":"rich","hl_at":"clean one point oh","footer":"card_limits: “Quantity, transaction, or merchant restrictions” · not_for: “Purpose, eligibility, or setup”"},"start":73.932,"dur":22.997,"words":[{"text":"Habit","s":74.232,"e":74.552},{"text":"two:","s":74.632,"e":75.352},{"text":"describe","s":75.432,"e":75.992},{"text":"the","s":76.032,"e":76.072},{"text":"options.","s":76.232,"e":77.152},{"text":"The","s":77.432,"e":77.512},{"text":"model","s":77.592,"e":77.832},{"text":"reads","s":77.912,"e":78.152},{"text":"both","s":78.232,"e":78.392},{"text":"the","s":78.432,"e":78.472},{"text":"option","s":78.632,"e":78.872},{"text":"names","s":79.032,"e":79.432},{"text":"and","s":79.512,"e":79.752},{"text":"their","s":79.832,"e":79.912},{"text":"descriptions.","s":80.032,"e":80.912},{"text":"For","s":81.112,"e":81.272},{"text":"options","s":81.352,"e":81.672},{"text":"that","s":81.752,"e":81.832},{"text":"get","s":81.912,"e":81.992},{"text":"confused,","s":82.152,"e":82.952},{"text":"say","s":83.032,"e":83.112},{"text":"what","s":83.192,"e":83.352},{"text":"each","s":83.432,"e":83.512},{"text":"one","s":83.592,"e":83.752},{"text":"covers,","s":83.832,"e":84.472},{"text":"what","s":84.552,"e":84.632},{"text":"it's","s":84.672,"e":84.872},{"text":"not","s":84.992,"e":85.112},{"text":"for,","s":85.192,"e":85.752},{"text":"and","s":85.832,"e":86.072},{"text":"give","s":86.152,"e":86.232},{"text":"an","s":86.272,"e":86.312},{"text":"example","s":86.552,"e":86.792},{"text":"or","s":86.872,"e":86.952},{"text":"two.","s":87.072,"e":87.672},{"text":"On","s":87.752,"e":87.832},{"text":"a","s":87.872,"e":87.912},{"text":"borderline","s":87.992,"e":88.472},{"text":"question","s":88.552,"e":88.952},{"text":"about","s":89.032,"e":89.272},{"text":"where","s":89.352,"e":89.432},{"text":"a","s":89.459,"e":89.485},{"text":"disposable","s":89.512,"e":90.152},{"text":"card","s":90.232,"e":90.472},{"text":"works,","s":90.672,"e":91.112},{"text":"plain","s":91.272,"e":91.432},{"text":"names","s":91.632,"e":91.832},{"text":"gave","s":91.872,"e":92.072},{"text":"point","s":92.152,"e":92.392},{"text":"nine.","s":92.552,"e":93.192},{"text":"Contrastive","s":93.352,"e":93.752},{"text":"descriptions","s":93.832,"e":94.472},{"text":"gave","s":94.552,"e":94.632},{"text":"a","s":94.712,"e":94.792},{"text":"clean","s":94.872,"e":95.192},{"text":"one","s":95.352,"e":95.432},{"text":"point","s":95.512,"e":95.672},{"text":"oh.","s":95.752,"e":96.312}]},{"template":"code","chapter":"Describe the options","narration":"In code, it looks like this. Each option gets the same fields, so the model can compare them side by side. You only need this for the pairs that actually get confused. Clear options can keep a one line description, or none at all.","params":{"file":"questions.py","lang":"py","lines":["\"topic\": Choice(","  instructions=\"Which disposable virtual card topic is the user asking about?\",","  criteria={","    \"get_card\": {\"what\": \"Purpose, eligibility, or setup\",","                 \"not_for\": \"Quantity, transaction, or merchant limits\",","                 \"examples\": [\"What are disposable cards for?\"]},","    \"card_limits\": {\"what\": \"Quantity, transaction, or merchant limits\",","                    \"not_for\": \"Purpose, eligibility, or setup\",","                    \"examples\": [\"Where can I use a disposable card?\"]},","  })"],"hl":[{"from":4,"to":9,"label":"same fields for every option","tone":"agent","at":"same fields"}]},"start":96.929,"dur":15.161,"words":[{"text":"In","s":97.229,"e":97.389},{"text":"code,","s":97.469,"e":98.029},{"text":"it","s":98.109,"e":98.189},{"text":"looks","s":98.269,"e":98.429},{"text":"like","s":98.509,"e":98.589},{"text":"this.","s":98.669,"e":99.269},{"text":"Each","s":99.549,"e":99.709},{"text":"option","s":99.789,"e":100.109},{"text":"gets","s":100.189,"e":100.349},{"text":"the","s":100.429,"e":100.589},{"text":"same","s":100.749,"e":100.909},{"text":"fields,","s":100.989,"e":101.549},{"text":"so","s":101.629,"e":101.709},{"text":"the","s":101.789,"e":101.829},{"text":"model","s":101.869,"e":102.109},{"text":"can","s":102.189,"e":102.269},{"text":"compare","s":102.349,"e":102.669},{"text":"them","s":102.749,"e":102.829},{"text":"side","s":102.909,"e":103.069},{"text":"by","s":103.149,"e":103.309},{"text":"side.","s":103.509,"e":103.949},{"text":"You","s":104.189,"e":104.269},{"text":"only","s":104.349,"e":104.589},{"text":"need","s":104.669,"e":104.829},{"text":"this","s":104.909,"e":104.989},{"text":"for","s":105.069,"e":105.149},{"text":"the","s":105.189,"e":105.229},{"text":"pairs","s":105.309,"e":105.549},{"text":"that","s":105.669,"e":105.789},{"text":"actually","s":106.029,"e":106.269},{"text":"get","s":106.349,"e":106.429},{"text":"confused.","s":106.589,"e":107.549},{"text":"Clear","s":107.709,"e":107.869},{"text":"options","s":107.949,"e":108.349},{"text":"can","s":108.429,"e":108.509},{"text":"keep","s":108.589,"e":108.669},{"text":"a","s":108.749,"e":108.909},{"text":"one","s":108.989,"e":109.069},{"text":"line","s":109.149,"e":109.229},{"text":"description,","s":109.309,"e":110.189},{"text":"or","s":110.309,"e":110.429},{"text":"none","s":110.589,"e":110.669},{"text":"at","s":110.749,"e":110.829},{"text":"all.","s":110.989,"e":111.469}]},{"template":"statement","chapter":"Give the full list","narration":"Habit three: give the full list. Jev accepts up to two hundred fifty five options, and each one costs only a few tokens. Don't hand-pick a shortlist; offer every team, every category. For deep taxonomies with thousands of leaves, go level by level, which we'll cover in the composition module.","params":{"eyebrow":"Up to 255 options per Choice","text":"Offer *every* option. A shortlist is a decision you made without the model.","art":"list","tone":"agent","punch":"every team"},"start":112.09,"dur":21.561,"words":[{"text":"Habit","s":112.39,"e":112.71},{"text":"three:","s":112.79,"e":113.59},{"text":"give","s":113.67,"e":113.91},{"text":"the","s":113.99,"e":114.07},{"text":"full","s":114.31,"e":114.55},{"text":"list.","s":114.63,"e":115.19},{"text":"Jev","s":115.67,"e":115.91},{"text":"accepts","s":116.11,"e":116.35},{"text":"up","s":116.39,"e":116.47},{"text":"to","s":116.55,"e":116.71},{"text":"two","s":116.79,"e":116.95},{"text":"hundred","s":117.03,"e":117.27},{"text":"fifty","s":117.51,"e":117.75},{"text":"five","s":117.91,"e":118.15},{"text":"options,","s":118.31,"e":119.03},{"text":"and","s":119.11,"e":119.19},{"text":"each","s":119.27,"e":119.43},{"text":"one","s":119.51,"e":119.67},{"text":"costs","s":119.75,"e":120.15},{"text":"only","s":120.27,"e":120.39},{"text":"a","s":120.47,"e":120.55},{"text":"few","s":120.63,"e":120.71},{"text":"tokens.","s":120.79,"e":121.59},{"text":"Don't","s":121.83,"e":122.07},{"text":"hand-pick","s":122.23,"e":122.71},{"text":"a","s":122.763,"e":122.817},{"text":"shortlist;","s":122.87,"e":123.75},{"text":"offer","s":123.95,"e":124.15},{"text":"every","s":124.23,"e":124.55},{"text":"team,","s":124.63,"e":125.19},{"text":"every","s":125.27,"e":125.51},{"text":"category.","s":125.59,"e":126.47},{"text":"For","s":126.71,"e":126.79},{"text":"deep","s":126.87,"e":127.11},{"text":"taxonomies","s":127.27,"e":127.83},{"text":"with","s":127.95,"e":128.07},{"text":"thousands","s":128.15,"e":128.55},{"text":"of","s":128.63,"e":128.71},{"text":"leaves,","s":128.79,"e":129.35},{"text":"go","s":129.43,"e":129.59},{"text":"level","s":129.75,"e":129.99},{"text":"by","s":130.07,"e":130.15},{"text":"level,","s":130.23,"e":130.79},{"text":"which","s":130.91,"e":131.03},{"text":"we'll","s":131.07,"e":131.19},{"text":"cover","s":131.27,"e":131.43},{"text":"in","s":131.51,"e":131.59},{"text":"the","s":131.617,"e":131.643},{"text":"composition","s":131.67,"e":132.23},{"text":"module.","s":132.31,"e":133.03}]},{"template":"chart","chapter":"Option order","narration":"Habit four: watch the order. TypeSafe's docs warn that Jev can lean toward the first option. So we reversed the order of all seventy seven intents and reran 3,080 messages. Four point nine percent of answers changed. But look where: thirty six percent of answers under point five flipped, and not a single answer above point nine.","params":{"title":"Answers that changed when the 77 options were reversed","kind":"bar","ymax":0.42,"data":[{"x":"p < 0.5","y":0.357,"label":"36%","tone":"bad","at":"thirty six"},{"x":"0.5 – 0.7","y":0.231,"label":"23%"},{"x":"0.7 – 0.9","y":0.048,"label":"5%"},{"x":"0.9 – 0.99","y":0,"label":"0%","at":"not a single"},{"x":"≥ 0.99","y":0,"label":"0%","hl":true}],"src":"BANKING77, jev-1.13.0, 3,079 messages × 2 orders, measured 2026-10-15. Overall 150 flips (4.9%)."},"start":133.651,"dur":24.016,"words":[{"text":"Habit","s":133.951,"e":134.271},{"text":"four:","s":134.431,"e":135.151},{"text":"watch","s":135.311,"e":135.551},{"text":"the","s":135.711,"e":135.791},{"text":"order.","s":135.951,"e":136.591},{"text":"TypeSafe's","s":136.831,"e":137.391},{"text":"docs","s":137.591,"e":137.791},{"text":"warn","s":137.871,"e":138.111},{"text":"that","s":138.151,"e":138.191},{"text":"Jev","s":138.351,"e":138.591},{"text":"can","s":138.711,"e":138.831},{"text":"lean","s":138.951,"e":139.071},{"text":"toward","s":139.111,"e":139.311},{"text":"the","s":139.431,"e":139.551},{"text":"first","s":139.711,"e":139.871},{"text":"option.","s":139.951,"e":140.831},{"text":"So","s":140.991,"e":141.151},{"text":"we","s":141.391,"e":141.471},{"text":"reversed","s":141.551,"e":141.951},{"text":"the","s":142.031,"e":142.111},{"text":"order","s":142.231,"e":142.351},{"text":"of","s":142.471,"e":142.591},{"text":"all","s":142.751,"e":142.911},{"text":"seventy","s":142.991,"e":143.271},{"text":"seven","s":143.311,"e":143.551},{"text":"intents","s":143.671,"e":144.191},{"text":"and","s":144.271,"e":144.351},{"text":"reran","s":144.431,"e":144.911},{"text":"3,080","s":144.991,"e":145.951},{"text":"messages.","s":146.031,"e":147.111},{"text":"Four","s":147.151,"e":147.311},{"text":"point","s":147.471,"e":147.711},{"text":"nine","s":147.791,"e":147.871},{"text":"percent","s":148.031,"e":148.271},{"text":"of","s":148.351,"e":148.431},{"text":"answers","s":148.631,"e":148.831},{"text":"changed.","s":148.911,"e":149.791},{"text":"But","s":149.831,"e":149.871},{"text":"look","s":149.951,"e":150.031},{"text":"where:","s":150.111,"e":150.911},{"text":"thirty","s":150.991,"e":151.311},{"text":"six","s":151.391,"e":151.551},{"text":"percent","s":151.631,"e":151.951},{"text":"of","s":151.991,"e":152.031},{"text":"answers","s":152.231,"e":152.431},{"text":"under","s":152.511,"e":152.751},{"text":"point","s":152.911,"e":153.071},{"text":"five","s":153.151,"e":153.391},{"text":"flipped,","s":153.631,"e":154.458},{"text":"and","s":154.484,"e":154.511},{"text":"not","s":154.591,"e":154.751},{"text":"a","s":154.831,"e":154.991},{"text":"single","s":155.071,"e":155.391},{"text":"answer","s":155.471,"e":155.791},{"text":"above","s":155.871,"e":156.191},{"text":"point","s":156.311,"e":156.431},{"text":"nine.","s":156.631,"e":157.071}]},{"template":"statement","chapter":"Option order","narration":"That's the pattern you'll see all through this course. When the model is sure, small changes don't move it. When it isn't, they do, and the probability tells you which case you're in. So route low confidence answers to review, and for close calls you care about, ask again with the options shuffled.","params":{"text":"Low probability is where answers are *fragile*. The number tells you which case you're in.","art":"snap","mood":"sure","react":{"at":"When the model is sure","mood":"sure"},"punch":"which case"},"start":157.667,"dur":18.739,"words":[{"text":"That's","s":157.967,"e":158.287},{"text":"the","s":158.327,"e":158.367},{"text":"pattern","s":158.447,"e":158.687},{"text":"you'll","s":158.767,"e":158.927},{"text":"see","s":159.087,"e":159.247},{"text":"all","s":159.367,"e":159.487},{"text":"through","s":159.567,"e":159.727},{"text":"this","s":159.807,"e":159.887},{"text":"course.","s":159.967,"e":160.767},{"text":"When","s":160.847,"e":160.927},{"text":"the","s":160.967,"e":161.007},{"text":"model","s":161.087,"e":161.327},{"text":"is","s":161.407,"e":161.487},{"text":"sure,","s":161.567,"e":162.127},{"text":"small","s":162.207,"e":162.447},{"text":"changes","s":162.527,"e":163.007},{"text":"don't","s":163.167,"e":163.327},{"text":"move","s":163.407,"e":163.567},{"text":"it.","s":163.647,"e":164.207},{"text":"When","s":164.247,"e":164.287},{"text":"it","s":164.367,"e":164.447},{"text":"isn't,","s":164.567,"e":165.087},{"text":"they","s":165.167,"e":165.327},{"text":"do,","s":165.487,"e":165.967},{"text":"and","s":166.047,"e":166.127},{"text":"the","s":166.167,"e":166.207},{"text":"probability","s":166.287,"e":166.927},{"text":"tells","s":166.967,"e":167.167},{"text":"you","s":167.247,"e":167.407},{"text":"which","s":167.527,"e":167.647},{"text":"case","s":167.767,"e":167.887},{"text":"you're","s":167.967,"e":168.047},{"text":"in.","s":168.127,"e":168.847},{"text":"So","s":168.967,"e":169.087},{"text":"route","s":169.247,"e":169.487},{"text":"low","s":169.647,"e":169.807},{"text":"confidence","s":169.887,"e":170.287},{"text":"answers","s":170.367,"e":170.687},{"text":"to","s":170.767,"e":170.807},{"text":"review,","s":170.847,"e":171.567},{"text":"and","s":171.687,"e":171.807},{"text":"for","s":171.887,"e":171.967},{"text":"close","s":172.047,"e":172.287},{"text":"calls","s":172.367,"e":172.527},{"text":"you","s":172.607,"e":172.687},{"text":"care","s":172.767,"e":172.927},{"text":"about,","s":173.007,"e":173.567},{"text":"ask","s":173.647,"e":173.887},{"text":"again","s":173.967,"e":174.367},{"text":"with","s":174.407,"e":174.447},{"text":"the","s":174.527,"e":174.607},{"text":"options","s":174.687,"e":175.007},{"text":"shuffled.","s":175.087,"e":175.807}]},{"template":"quiz","chapter":"Quick check","narration":"Quick check. A support bot classifies messages into five departments. A message arrives about something none of them handle. What happens if the Choice has no other option?","params":{"q":"A message fits **none** of your five departments, and there's no “other” option. What happens?","options":["The API returns an error","The model returns an empty answer","The model must spread probability over your five options and pick one","The model invents a sixth department"],"answer":2,"explain":"A Choice only scores the options you give it, so it has to pick one, usually with a flat, low-confidence distribution. Add “other” so it has somewhere honest to go."},"start":176.406,"dur":15.949,"words":[{"text":"Quick","s":176.706,"e":176.946},{"text":"check.","s":177.026,"e":177.546},{"text":"A","s":177.746,"e":177.826},{"text":"support","s":178.026,"e":178.226},{"text":"bot","s":178.306,"e":178.546},{"text":"classifies","s":178.706,"e":179.186},{"text":"messages","s":179.466,"e":179.746},{"text":"into","s":179.906,"e":180.066},{"text":"five","s":180.186,"e":180.306},{"text":"departments.","s":180.386,"e":181.426},{"text":"A","s":181.466,"e":181.506},{"text":"message","s":181.586,"e":181.906},{"text":"arrives","s":182.026,"e":182.306},{"text":"about","s":182.426,"e":182.546},{"text":"something","s":182.626,"e":183.026},{"text":"none","s":183.106,"e":183.186},{"text":"of","s":183.266,"e":183.346},{"text":"them","s":183.426,"e":183.506},{"text":"handle.","s":183.586,"e":184.386},{"text":"What","s":184.506,"e":184.626},{"text":"happens","s":184.706,"e":184.946},{"text":"if","s":185.026,"e":185.106},{"text":"the","s":185.186,"e":185.266},{"text":"Choice","s":185.426,"e":185.586},{"text":"has","s":185.666,"e":185.906},{"text":"no","s":185.986,"e":186.146},{"text":"other","s":186.266,"e":186.386},{"text":"option?","s":186.466,"e":187.106}]},{"template":"exercise","chapter":"Try it","narration":"Try it. Have your agent audit every Choice in your code for the four habits.","params":{"title":"Audit your Choices","intro":"Paste this into your coding agent:","prompt":"Review every multiple-choice decision we ask a model to make. For each:\n1. Is the option list complete? Add an explicit \"other\" / none option\n   if inputs can fall outside it.\n2. Does each option have a one-line description? For options that are\n   often confused, add contrastive what / not_for / examples fields.\n3. Does code treat low-confidence answers differently?\n4. Add a test that re-asks 20 known close calls with the options\n   shuffled and reports any answer that changes."},"start":192.355,"dur":5.783,"words":[{"text":"Try","s":192.655,"e":192.975},{"text":"it.","s":193.055,"e":193.295},{"text":"Have","s":193.615,"e":193.775},{"text":"your","s":193.855,"e":193.935},{"text":"agent","s":194.015,"e":194.335},{"text":"audit","s":194.535,"e":194.735},{"text":"every","s":194.815,"e":195.135},{"text":"Choice","s":195.215,"e":195.455},{"text":"in","s":195.535,"e":195.575},{"text":"your","s":195.615,"e":195.695},{"text":"code","s":195.775,"e":196.095},{"text":"for","s":196.135,"e":196.175},{"text":"the","s":196.255,"e":196.335},{"text":"four","s":196.415,"e":196.655},{"text":"habits.","s":196.735,"e":197.535}]},{"template":"recap","chapter":"Recap","narration":"So: a Choice gives you a distribution, and its confidence says how peaked it is. Offer a way out. Describe the options that get confused. Give the full list. And remember that low confidence answers are the fragile ones. Next: the Score.","params":{"items":["A Choice returns a probability for every option, plus confidence.","Always offer “other” when inputs can fall outside your list.","Describe confusable options: what, not_for, examples.","Low-confidence answers are fragile: review or re-ask shuffled."],"next":"Score: a position on a scale you describe","next_at":"Next"},"start":198.138,"dur":18.906,"words":[{"text":"So:","s":198.438,"e":199.158},{"text":"a","s":199.238,"e":199.318},{"text":"Choice","s":199.518,"e":199.718},{"text":"gives","s":199.798,"e":199.958},{"text":"you","s":199.998,"e":200.038},{"text":"a","s":200.118,"e":200.158},{"text":"distribution,","s":200.198,"e":201.238},{"text":"and","s":201.278,"e":201.318},{"text":"its","s":201.438,"e":201.558},{"text":"confidence","s":201.878,"e":202.198},{"text":"says","s":202.318,"e":202.438},{"text":"how","s":202.558,"e":202.678},{"text":"peaked","s":202.718,"e":202.998},{"text":"it","s":203.038,"e":203.078},{"text":"is.","s":203.158,"e":203.718},{"text":"Offer","s":203.958,"e":204.278},{"text":"a","s":204.318,"e":204.358},{"text":"way","s":204.438,"e":204.598},{"text":"out.","s":204.678,"e":205.198},{"text":"Describe","s":205.398,"e":205.878},{"text":"the","s":205.958,"e":206.038},{"text":"options","s":206.278,"e":206.518},{"text":"that","s":206.598,"e":206.678},{"text":"get","s":206.758,"e":206.838},{"text":"confused.","s":206.998,"e":207.878},{"text":"Give","s":207.958,"e":208.118},{"text":"the","s":208.198,"e":208.438},{"text":"full","s":208.558,"e":208.678},{"text":"list.","s":208.758,"e":209.638},{"text":"And","s":209.678,"e":209.718},{"text":"remember","s":209.798,"e":210.278},{"text":"that","s":210.398,"e":210.518},{"text":"low","s":210.678,"e":210.838},{"text":"confidence","s":210.918,"e":211.398},{"text":"answers","s":211.478,"e":211.878},{"text":"are","s":211.958,"e":212.038},{"text":"the","s":212.118,"e":212.198},{"text":"fragile","s":212.278,"e":212.678},{"text":"ones.","s":212.758,"e":213.718},{"text":"Next:","s":213.798,"e":214.518},{"text":"the","s":214.598,"e":214.678},{"text":"Score.","s":214.758,"e":215.478}]}],"total":217.444,"audio":"/audio/06-choice.mp3"}