Most of what we spend on models goes to tests. That makes the testing method a cost decision before it is a quality decision, and our first rule is to keep the two kinds of test apart.
Two kinds of test
Plumbing tests check that the code works: a pitch gets created, a draft gets saved, a failed call retries. They run on the cheapest model that exercises the code, and with the fewest rounds that exercise it. Quality tests and benchmarks run on the models we might ship, and they run sparingly. The rule came from one run. An end-to-end run on our most expensive model, with five rounds of plan feedback where two would have proved the code, cost $5.73 and used up that day's whole budget.
Every model call also goes through one logged door that records the tenant, the model, the tokens and the cost, and enforces a daily cap with a kill switch. We cannot reduce a number we cannot see.
How we ran a blind round
- One frozen brief. Every model got the same input, captured from a real run.
- One draft per model. Two drafts per model would double the bill and add little, because the models differ by more than a single draft's luck.
- Letters, not names. The drafts were labelled A to E, and the key stayed sealed in a file with a hash until the ranking was done.
- A human ranks. The founder read every draft and ranked them best to worst, with his reasons.
- Claude's opinion does not count. We asked Claude to rate the same drafts blind. It put the Anthropic model first both times and Qwen last both times, while the founder put Qwen first or tied for first. A model rating models tends to prefer the style it was trained on. We discarded those ratings.
Four rounds, and the pick
| Round | Models | Founder's order |
|---|---|---|
| 1 | Opus 5.5, GPT-5.6 Sol, Sonnet 5.5 | Opus ($0.23), Sol ($0.13), Sonnet ($0.11) |
| 2 | Sol, DeepSeek V4 Pro, Opus, Sonnet | Sol ($0.15), DeepSeek ($0.08), Opus ($0.21), Sonnet ($0.10) |
| 3 | Qwen 3.7 Plus joins | Qwen ($0.018) and Opus ($0.15) tied for first |
| 4 | Qwen, Sonnet, Opus, Sol | Sonnet ($0.09) and Qwen ($0.02) tied for first, Opus third, Sol last |
Averaged over the rounds, Qwen's place was 1.5, Opus 2.1, Sol 2.5, Sonnet 3.3 and DeepSeek 3.3. Qwen had the best average of any model we tried. In round 3 the founder called it against Opus a coin toss, with Qwen slightly ahead. It matched the best model on quality, and it did so at about a tenth of the price. The price was a result of the test, and quality decided the pick. We merged Qwen as the Writer on October 17.
We ran a six-model blind test on the strategy agent the same way. Opus came first at $0.21 a run. Our most expensive model came second at $0.74 on production runs, and everything landed within about ten percent of the best. Two of the other models spent their entire 22,000-token limit on reasoning and returned nothing until we raised the limit to 64,000. The one with the lowest sticker price, $0.04, took 585 seconds.
The retest
Nine days later we wanted to know whether a heavier writing prompt would help. We ran eighteen drafts on one new brief: three models (Opus, Sol, Qwen) against three prompts, a plain control, a light version and a heavy version with a long section on the shape of prose. It cost $1.52 in all, at about $0.02 for each Qwen draft, $0.08 for each Sol draft and $0.13 for each Opus draft.
The founder ranked each of six rounds blind. This time Opus finished first or tied for first in every round he rated, Sol came second, and Qwen came third. The heavy prompt hurt Qwen the most: its sentences dropped to 8.5 words on average, and 93 percent of its six-sentence runs were flagged as one repeated beat. On Opus, the heavy prompt pushed "you" up to three or four per hundred words. The plain control won or tied for every model.
Two things in that result surprised us. Our own odds were wrong again, since we had expected the heavy prompt to help Qwen most. And the order of the first four rounds did not carry over. The brief and the prompts were new, and the founder was by then reading for habits he had not named in the first rounds, such as the same subject opening sentence after sentence. A winner is the winner of its test, and this was a different test.
What we changed
We moved the Writer and the Pitcher to Opus 5.5. A draft now costs about six times what it did. Our target is ten cents or less per write, and an Opus draft in these tests cost 13 to 19 cents, so the bill is a standing problem and not a settled one. Qwen stays in our tests, and the next time the brief or the prompt changes, it runs in the blind round again.
What we would repeat
- Seal the key before anyone reads a draft.
- Run one draft per arm and spend the saved money on more arms.
- Retest a decision when the conditions move. A winner is the winner of that test, and the loser of one test can win the next.
- Keep the cheapest model on the plumbing. A cheap test model can cause a bug of its own, and in our experience that is rare and cheap to find.
- Put price next to quality, never in front of it. Qwen won on quality, and its price made the win easy to take.