Skip to content
Method

How we measured the race

The race on the home page replays real measurements. Here is what we ran, on what, and how the numbers are computed, so you can check them or run the same test on your own cases.

  • 99×faster

    Than GPT-6.1 Astra, the fastest LLM we measured (1.65 s median). Against the others the gap is larger.

  • 96×cheaper

    Than Claude Haiku 4.5, the cheapest, at list price checked 2026-10-12.

  • 17 msper decision

    VAKYN's median wall-clock time for one call, measured from the machine that ran the test.

  • 72decisions

    Yes-or-no, pick-one and score questions on documents of about 1,220 characters, each asked 3 times per model.

The decisions

72 decisions: yes-or-no, pick-one and score questions about support emails, expense reports, product reviews and other short documents we wrote for VAKYN's examples (about 1,220 characters each). No public benchmark questions. Each model saw each decision 3 times.

The method

  • One decision per call, calls one after another, so each number is the time a single decision takes.
  • Time is the median wall-clock time of a call, measured from the machine that ran the test.
  • Cost is the tokens each provider reported for the call, at its list price on the date shown.
  • The language models got the same document and question, with the allowed answers listed, and had to reply with JSON. They ran with their fastest settings: thinking off where it can be turned off, the lowest reasoning effort otherwise.
  • Jev appears with its published figures; we did not run it.

Recorded method: one decision per call, calls in sequence, wall-clock time from this machine; cost = what OpenRouter charged, or usage tokens × list price; VAKYN: 16.6 ms median, one RTX 5090; the LLMs through OpenRouter.

What this does not say

Speed and cost say nothing about how often it is right. A decision model is right for decisions you make many times a day, where it is measured on your own cases; for open questions, use a language model. Times depend on the document length and on the distance to each provider.