Routing by confidence
Let your code act on the answers the model is sure about and send the rest to a person or a larger model.
When to use it
- Most cases are routine, but a wrong decision costs something: a refund paid, a review removed.
- Someone already checks these cases by hand, and you want them to check fewer.
- You call a large language model on every case today and want to call it only for the hard ones.
How it works
Every answer comes with how sure it is. A choice and a score carry a confidence; for a noul it is how far the probability leans either way, max(p, 1 - p). See Confidence for what the numbers mean. Pick a threshold, and act only when every answer in the request clears it.
Request
State + questions
- a review, a ticket, a report
Answers
Each with how sure it is
- fake 0.97
- topic 0.91
- negative 0.62
lowest ≥ threshold
Your code acts on the answers.
lowest < threshold
A person, or a larger model, decides.
from typesafe_sdk import TypeSafeClient # Reads TYPESAFE_BASE_URL (https://api.vakyn.com) and TYPESAFE_API_KEY from the environment.client = TypeSafeClient()THRESHOLD = 0.70 # set it from what a wrong answer costs you def sureness(result) -> dict[str, float]: """How sure each answer is, from 0 to 1.""" sure = {name: max(a.noul, 1 - a.noul) for name, a in result.nouls.items()} sure |= {name: a.confidence for name, a in result.choices.items()} sure |= {name: a.confidence for name, a in result.scores.items()} return sure def decide(state, questions): result = client.system_one(state=state, questions=questions) sure = sureness(result) weakest = min(sure, key=sure.get) if sure[weakest] >= THRESHOLD: return "act", result # every answer is sure enough: your code acts return "review", result # a person (or a larger model) takes this one- Measure how many cases fall under the threshold. If it is too many, the options are probably unclear or incomplete; fix the questions before lowering the bar.
- Log the whole answer, not only the pick. When a person corrects a case, you can see whether the model had the right answer in second place.
- Start strict and lower the threshold as you see how often the sure answers are right. The Confidence page shows how to pick it from your own data.
Example: moderating product reviews
Each review gets a noul (fake?), a choice (topic) and a score (how negative). The code removes or publishes only when every answer is sure enough; anything else waits for a moderator. Step by step: How VAKYN works.
Questionsfakenoultopicchoicenegativescore
Clear case
Acts automaticallyStopped working after five weeks. 1 star. I bought the Tallow & Pine CafeDuo 2 in March and used it every morning for two coffees. In week five the steam wand stopped heating, then the pump started rattling and the machine shut itself off mid-shot. I descaled it twice with the brand's own tablets and followed the reset steps in the manual exactly. Nothing helped. For a machine at this price I expected years, not weeks. Do not buy. Marta K., verified purchase
Question Answer Confidence fake yes 0.04 0.96 topic quality 0.97 negative 3.94 of 4 0.95 Every answer is at least 0.70 sure (lowest: negative 0.95), so your code acts on it.
Borderline case
Goes to a personDecent kettle for what I paid. It took ages to show up and the box was a bit crushed, but it boils fine. Would maybe buy again if it goes on sale. J.
Question Answer Confidence fake yes 0.03 0.97 topic quality 0.23 negative 1.93 of 4 0.94 topic is only 0.23 sure, under 0.70, so a person decides.
Clear case
curl -s "https://api.vakyn.com/v1/systemone" \ -H "Authorization: Bearer $TYPESAFE_API_KEY" \ -H "Content-Type: application/json" \ -o answer.json \ -d @- <<'JSON'{ "model": "jev-latest", "state": "Stopped working after five weeks. 1 star.\n\nI bought the Tallow & Pine CafeDuo 2 in March and used it every morning for two coffees. In week five the steam wand stopped heating, then the pump started rattling and the machine shut itself off mid-shot. I descaled it twice with the brand's own tablets and followed the reset steps in the manual exactly. Nothing helped. For a machine at this price I expected years, not weeks. Do not buy.\n\nMarta K., verified purchase", "questions": { "fake": { "type": "noul", "instructions": "Is this review likely fake: paid for, written by the seller, or not based on using the product?" }, "topic": { "type": "choice", "instructions": "What is the review mainly about?", "criteria": { "quality": "How the product works, feels or holds up", "delivery": "Shipping, packaging and arrival", "price": "Price and value for money", "service": "Support, returns and refunds", "other": null } }, "negative": { "type": "score", "instructions": "How negative is the review?", "criteria": [ "Positive", "Mostly positive", "Mixed", "Mostly negative", "Very negative" ] } }}JSON # Act when every answer is at least 0.70 sure; otherwise a person decides.jq -r '.answers as $a | {fake: ([$a.fake.noul, 1 - $a.fake.noul] | max), topic: $a.topic.confidence, negative: $a.negative.confidence} | to_entries | min_by(.value) as $weakest | if $weakest.value >= 0.70 then "act: \(if $a.fake.noul >= 0.5 then "remove" else "publish" end), topic \($a.topic.choice), negative \($a.negative.score) of 4" else "human: \($weakest.key) is only \($weakest.value * 100 | round / 100) sure" end' answer.jsonThe borderline case runs the same code with its own state. topic is only 0.23 sure, under 0.70, so it prints a human line and a person decides. Change the threshold to match what a wrong answer costs you.
Example: approving expense reports
The same rule on structured data. The report goes in as JSON with the policy next to it; a choice says approve, flag or reject, a noul says whether any line breaks the policy, and a score rates the risk of paying it as filed. Reports the model is sure about are paid or returned by code; the rest go to finance.
Questionsdecisionchoiceover_policynoulriskscore
Clear case
Acts automatically{ "policy": { "rail": "Second class, booked through the travel desk", "hotel": "Up to EUR 180 a night", "meals": "Up to EUR 60 a day while travelling, no alcohol", "receipts": "A receipt for every line over EUR 25" }, "report": { "employee": "Ines Varga, field engineer", "trip": "Customer site visit in Lyon, 6 October 2026", "lines": [ { "date": "2026-10-06", "item": "Flight, Brussels to Lyon and back, business class, booked privately", "amount_eur": 1240, "receipt": false }, { "date": "2026-10-06", "item": "Hotel, three nights for a one-day visit", "amount_eur": 1260, "receipt": false }, { "date": "2026-10-06", "item": "Dinner, alone, two bottles of wine", "amount_eur": 310, "receipt": false } ], "total_eur": 2810 }}
Question Answer Confidence decision reject 0.82 over_policy yes 1.00 1.00 risk 2.97 of 3 0.97 Every answer is at least 0.70 sure (lowest: decision 0.82), so your code acts on it.
Borderline case
Goes to a person{ "policy": { "rail": "Second class, booked through the travel desk", "hotel": "Up to EUR 180 a night", "meals": "Up to EUR 60 a day while travelling, no alcohol", "receipts": "A receipt for every line over EUR 25" }, "report": { "employee": "Pieter Claes, account manager", "trip": "Trade fair in Milan, 13 to 14 October 2026", "lines": [ { "date": "2026-10-13", "item": "Rail, Brussels to Milan, second class, booked by the travel desk", "amount_eur": 164, "receipt": true }, { "date": "2026-10-13", "item": "Hotel, one night, city tax EUR 9 included", "amount_eur": 189, "receipt": true }, { "date": "2026-10-13", "item": "Dinner with two people from a prospect", "amount_eur": 118, "receipt": true, "note": "Talked about a pilot" }, { "date": "2026-10-14", "item": "Taxi to the fair", "amount_eur": 27, "receipt": false, "note": "Driver's card reader was broken" } ], "total_eur": 498 }}
Question Answer Confidence decision reject 0.65 over_policy yes 0.94 0.94 risk 2.52 of 3 0.52 risk is only 0.52 sure, under 0.70, so a person decides.
Clear case
curl -s "https://api.vakyn.com/v1/systemone" \ -H "Authorization: Bearer $TYPESAFE_API_KEY" \ -H "Content-Type: application/json" \ -o answer.json \ -d @- <<'JSON'{ "model": "jev-latest", "state": { "policy": { "rail": "Second class, booked through the travel desk", "hotel": "Up to EUR 180 a night", "meals": "Up to EUR 60 a day while travelling, no alcohol", "receipts": "A receipt for every line over EUR 25" }, "report": { "employee": "Ines Varga, field engineer", "trip": "Customer site visit in Lyon, 6 October 2026", "lines": [ { "date": "2026-10-06", "item": "Flight, Brussels to Lyon and back, business class, booked privately", "amount_eur": 1240, "receipt": false }, { "date": "2026-10-06", "item": "Hotel, three nights for a one-day visit", "amount_eur": 1260, "receipt": false }, { "date": "2026-10-06", "item": "Dinner, alone, two bottles of wine", "amount_eur": 310, "receipt": false } ], "total_eur": 2810 } }, "questions": { "decision": { "type": "choice", "instructions": "What should finance do with this expense report?", "criteria": { "approve": "Pay it as filed", "flag": "Hold it until a person checks it", "reject": "Send it back to the employee to fix" } }, "over_policy": { "type": "noul", "instructions": "Does any line in the report break the expense policy?" }, "risk": { "type": "score", "instructions": "How risky is it to pay this report as filed?", "criteria": [ "No risk", "Low risk", "Some risk", "High risk" ] } }}JSON # Act when every answer is at least 0.70 sure; otherwise a person decides.jq -r '.answers as $a | $a | map_values(if .type == "noul" then [.noul, 1 - .noul] | max else .confidence end) | to_entries | min_by(.value) as $weakest | if $weakest.value >= 0.70 then "act: \($a.decision.choice), over policy \(if $a.over_policy.noul >= 0.5 then "yes" else "no" end), risk \($a.risk.score) of 3" else "human: \($weakest.key) is only \($weakest.value * 100 | round / 100) sure" end' answer.jsonThe borderline case runs the same code with its own state. risk is only 0.52 sure, under 0.70, so it prints a human line and a person decides. Change the threshold to match what a wrong answer costs you.