Decision models had their launch moment this autumn. Within a few weeks of the first one, dozens of similar models (opens in a new tab) appeared. They all promise the same thing: a typed answer with a probability instead of a paragraph, fast enough to put inside software.
Several of them are general 27-billion-parameter language models with a decision layer on top. That works, and the answers can be good. It also undoes much of what makes a decision model worth having.
What a decision model is for
A decision model is the fast part of a system. Psychologists call it System 1: the quick, automatic judgement you use for the everyday cases, while slower, careful thinking handles the hard ones. In software, that means answering the same kind of question thousands of times a day (is this phishing, which team gets this ticket, approve or reject) in milliseconds and at a cost per decision close to nothing.
Speed and cost are the whole point. A decision model that takes a fifth of a second and a data-center GPU per answer has given up most of what it was for.
The arithmetic of size
A decision model reads the document once and doesn't generate text, so almost all of its work is reading. Reading one token costs about two calculations per parameter (Kaplan et al., 2020 (opens in a new tab)). The cost of a decision therefore grows in step with the model's size and the length of the document.
Memory follows the same line. A 27B model's weights take about 54 GB at 16-bit precision, a 4B model's about 8 GB. One published 27B decision model needs about 85 GB of GPU memory (opens in a new tab) at its full context. That is a data-center card per copy, where a small model fits on one card with room to spare. On a single RTX 6000-class card, our 4B model answers a short ticket in 16.6 ms.
What size buys on everyday decisions
For the decisions a model was trained on, very little. In our own runs, going from 4B to 9B adds about one point on everyday decisions. Our trained 0.8B model beats an untrained 35B general model on the same decisions. Training does more than size.
Others find the same. Smaller fine-tuned models consistently outperform larger zero-shot models (opens in a new tab) at text classification. In Red Hat's test of prompt-injection detectors (opens in a new tab), a 200M-parameter classifier scored 89.0% against 89.3% for a 35B model, at 54 ms instead of 313 ms. NVIDIA's researchers estimate that serving a 7B model is 10 to 30 times cheaper (opens in a new tab) than serving a 70B–175B one.
Where size does help
Size isn't useless. Bigger models do better on long, multi-step work: in our runs, the 9B is clearly ahead of the 4B on workflows where several checks over a long document lead to one action. Bigger models also do better on questions unlike anything they were trained on, and at noticing when a request is out of scope.
Training closes most of that gap. One open model family trained at four sizes, from 0.8B to 27B, scored about the same at every size above 0.8B on cases like its training data, and only pulled apart on cases it had never seen. So the useful question is what your decisions look like, and whether the model has been trained on them.
That is why we right-size instead of scaling up. Our largest model, the one behind VAKYN Cloud, is a 9B, used for the long multi-step cases. Everyday decisions run on the small models, and the VAKYN loop trains them on your own cases until the gap that matters to you is closed.
Small first, bigger when unsure
The cheapest way to get both speed and quality is a cascade. Every case goes to the small model first. It decides the clear ones in milliseconds and says how sure it is. The unsure ones go up to a bigger model or a person.
Most of the volume never touches the big model, so the bill and the latency are set by the small one, and the hard cases still get careful thinking. Intelligence thinks. Intuition decides, and intuition shouldn't need 27 billion parameters.
To see what a right-sized model does on your own cases, test it on your data.