Tune the question before you change the model

Parallel (opens in a new tab) put eight decision models on four tasks from their own production pipeline: reranking search results, spotting real changes on a page, filtering low-quality content, routing queries. Then they rewrote the question each model was asked, five rounds of it, and scored the winners again, which is where most comparisons stop. The table changed.

The rewrite moved more than the models did

Before any tuning the field was spread wide. On page-change classification the scores ran from 93.3 down to 23.5, which reads like a settled question about which models can do the job at all.

Then they tuned: "We tuned each model across 5 rounds on a small labeled calibration set (800 examples for page change), and scored the winning prompt once on a held-out set." No training and no new weights, only the wording of the question and the descriptions of the options.

Read the order. The three models that had ranked at the top gained the least: +1.0, +2.3 and +0.3. The ones that had looked worst gained the most. After tuning, a 9B model passed an untuned 26B, and a hosted API that had ranked fifth went past the untuned leader.

So part of what the first table measured was how well each model happened to suit the wording someone wrote first. A ranking of untuned prompts ranks first drafts.

Why wording moves a decision model so much

Parallel names the reason: "A fine-tuned classifier is fixed by inference time... A decision model, meanwhile, takes its instructions as input, so its behavior can be tuned."

That cuts both ways. Everything the model knows about your decision arrives in the request: what the question means, what each option covers, where the line between two of them falls. A classifier learned those lines from thousands of examples. A decision model has to be told, in a sentence.

Which is why a question like this one

"intent": {  "type": "choice",  "instructions": "Which code fits the customer's message?",  "criteria": {    "CRD": "CRD", "DSP": "DSP", "TRF": "TRF",    "FEE": "FEE", "ACC": "ACC", "OTH": "OTH"  }}

is not a hard question the model is failing. It is an unasked one. Six strings, with no meaning attached to any of them. Whatever it scores, it scores by guessing what a bank means by DSP.

What happened when we tried it on ours

We ran the same loop against vakyn-fast and vakyn-max on three public sets neither model has been trained on: TREC question types, which is the question above almost exactly (six codes, nothing said about any of them), Banking77 intents (77 options named in snake_case and described nowhere), and tweets labeled for irony. Each set was cut in half, one half to work on and one half held back and scored once, at the end.

One round on two of them, two rounds on TREC. The rules were written by hand, the way the skill has an agent write them: read the biggest group of mistakes, add one sentence, score it, keep it only if the score went up. All of it together came to about 8,700 requests and forty minutes.

Four things came out of it.

1. Undescribed options are the biggest lever. TREC hands you six codes: ABBR, DESC, ENTY, HUM, LOC, NUM. Writing one line for each took vakyn-fast from 121 of 250 held-back cases right to 202. Banking77, with 77 snake_case names, went from 178 of 300 to 206.
2. A tuned small model can pass an untuned big one. On irony, tuned vakyn-fast reached 0.889 average precision against 0.875 for untuned vakyn-max. Parallel saw the same between a 9B and a 26B. If you were about to move up a model size, rewrite the question first and find out whether you still need to.
3. The gap between the two halves is the warning. Our second TREC round reached 224 of 250 on the half we were tuning on and 202 on the half we had not touched. That rewrite had quoted the cases we had just read, and the held-back number is the one that was true.
4. The best wording was not the same for both models. The rule that gained vakyn-fast 17 cases cost vakyn-max four, so the loop kept a different final wording for each. Four cases on the tuning half is thin evidence, so take it as a reason to tune per model rather than as a law.

The skill's own support-ticket example kept the probabilities as well. With bare codes the median confidence was 0.27, and after the rewrite 0.85. A question nobody has described shows up in the confidence before it shows up in the score, so you can see it coming without labeling anything.

All of that is one run, a 50/50 split, rules written by hand rather than by the shipped skill, and small sets. The size of each gain belongs to that question, not to yours. The direction is the same one Parallel found: the worse the question was written, the more of the gap was wording.

So we made it a skill

Most of that work was mechanical. Group the misses by confusion pair, write a sentence, re-score, keep or discard, and never look at the held-back half. Done by hand it also gives you three ways to fool yourself: choosing the split after you have seen the scores, reading the held-back cases "just to understand them", and keeping a rewrite because it felt better. So we wrote the loop down and shipped it as optimize, one of the VAKYN skills for Claude Code, Codex, Cursor and OpenCode. You give it one question, your labeled cases and a model, and it does the measuring while you or your agent write the rules.

The discipline is worth copying even if you never run ours.

• Split the cases before the first round, by a hash of the case id, so the split never moves.
• Score every round on the dev side only.
• Read the reserved side once, at the end, as counts, not as cases to read and learn from.
• Keep the rewrite only if it is better on dev and not worse on reserved.

Grouping the mistakes is what keeps the rules general. For a choice question, group by the pair: the right option and the one the model picked instead. In the bank example that ships with the skill, DSP -> TRF: 6 is not six problems. It is one. Nobody ever said where a payment the customer did not make ends and a transfer begins. Read three cases of the group, write one sentence, score it.

Three rules we hold to while writing them:

1. A rule states a pattern, never a copied case. That is the mistake our own second TREC round made: 224 of 250 on the half we tuned on, 202 on the half we had not.
2. Describe, do not rename. A bare code gets a line that says what belongs in it, and the key stays as it is, because your code branches on CRD, your logs store it and your thresholds are set per option.
3. No sums, dates or thresholds inside the question. "Over 30 days" is arithmetic. Work it out in your code and put the answer in the state.

Wording is tuned per model, so two models is two runs. And watch the size of your sets while you read the result: with 20 reserved cases one case is worth five points, so "not worse on reserved" is a weak bar. Say so when you report the number.

Where wording stops

The most useful number in Parallel's post is about the other direction. Their own small fine-tuned reranker scored 75.21 where the best decision model scored 71.45: "Our small fine-tuned reranker scored 3.8 percentage points above the best decision model, and a model that size costs far less to serve than any decision model we tested."

So there are two steps, and they are not interchangeable.

Read the misses to tell which step you need. If a group of them shares something you could write down in one sentence, it is wording, and it is free to fix. If the model gets a group wrong because it has never seen that kind of case (your document shape, your industry's vocabulary, the distinction your business cares about and nobody else does), no sentence will fix it, and every round you spend trying is a round spent fitting your dev set.

The second step is the VAKYN loop: we find where the model fails on your work, train exactly there, prove it on your cases, and repeat when your work changes.

Start with the first step. It costs an afternoon and a few hundred labels, the skill is free, and it is the only way to find out what is missing before you go shopping for a bigger model.

To see where a tuned question gets you on your own cases, test it on your data.