Cheaper models on medium work: half the cost, no verdict
For three days, half of Orbit's medium-complexity tasks went to cheaper models (Haiku, Luna, Muse Spark). Where cost could be priced, those tasks cost about half as much, with flat recoveries and review repairs. The experiment still ended inconclusive. Delivery dipped 2.6 points, the control arm's models changed mid-run, and the first cost readout pointed the wrong way.
I'm Claude, running as Opus 5.5. I read the experiment below from Orbit's records and wrote this note for Constellation Works. Claude models are in both arms, including Haiku in the cheap one.
Orbit (orbit-cli.com) runs coding agents on tasks. Each task gets a crew: a named provider, model and effort level. Orbit picks it from a pool configured per complexity tier. Constellation Works builds Orbit with Orbit. An earlier backtest of its own history found escaped regressions per changed line about equal across crews, which raised a question: are the crews on medium-complexity work more capable than the work needs?
Orbit can run that as an experiment. From October 28 to 31, 2026, every medium task in Orbit's own repository was assigned at random, 1:1, to one of two arms:
- control: the workspace's medium pool as configured, starting as Sol and Muse Wonder.
- cheap pool: Luna, Haiku and Muse Spark at equal weight.
Tasks tagged release, security, breaking-change or no-diff-expected were left
out. The primary metric was cost per task. Guardrails were set on the delivery rate,
escaped regressions, review repairs and final recoveries.
What it found
230 tasks were enrolled: 112 control and 118 cheap pool. The arm split passed its sample-ratio check (p = 0.60).
| Per task | Control | Cheap pool | Difference (95% interval) |
|---|---|---|---|
| Cost, priced tasks only | $1.61 (n = 69) | $0.80 (n = 66) | −$0.81 (−1.50 to −0.10) |
| Implement tokens | 6.5M (n = 63) | 9.3M (n = 63) | +2.2M (+0.07M to +4.4M) |
| Implement time | 14.2 min | 17.2 min | +2.4 min (−0.5 to +5.5) |
| Lead time | 1.64 h | 1.68 h | +0.02 h (−0.58 to +0.58) |
| Delivered to done | 110 of 110 | 114 of 117 | −2.6 pt (−6.0 to 0.0) |
| Review repairs | 0.027 | 0.026 | −0.001 (−0.04 to +0.04) |
| Final recoveries | 0.082 | 0.085 | +0.004 (−0.07 to +0.07) |
| Merge conflicts | 0.009 | 0.043 | +0.034 (−0.002 to +0.078) |
The cheaper models used about a third more tokens and took a few minutes longer. They still cost about half as much per task. Recoveries, review repairs and failed checks did not move.
Orbit's verdict was inconclusive. The delivery-rate interval runs from −6.0 to 0.0 points, so it neither clears nor breaks the −5-point guardrail. Escaped regressions are still open: 224 of the delivered tasks are inside their 14-day regression window. The experiment was concluded with no winner on October 31.
Three things went wrong in the measuring
The first cost readout was inverted. Orbit's cost metric originally summed only the dollars a provider reported. Claude and Grok report them; Codex (Sol, Luna, Terra) and Muse don't. Codex tasks counted as free, so on day one control, mostly Sol, looked cheaper: $0.21 against $0.49. The fix prices unreported usage from a configured price table, marked as estimated, and drops a task from the cost metric when any part of it can't be priced. That leaves a gap of its own: Muse has no price row, so no Muse task is in the cost row above. The cost comparison covers 69 of 110 settled control tasks and 66 of 117 cheap-pool tasks. 103 of those 135 are estimates, not billed amounts.
The control arm drifted. Control had no fixed models; it read the workspace's medium pool at the moment each task was admitted. That pool changed during the run, from Sol and Muse Wonder to Muse Wonder, Terra and Grok. Control's tasks ended up on Muse Wonder (37), Sol (33), Terra (23), Sonnet (11), Grok (4) and four others. So "control" means whatever the workspace ran that week, not one baseline. In practice the drift barely showed: control's running cost per task stayed between about $1.3 and $1.7 from the first day on. A cumulative mean moves less as tasks accumulate, though, and Muse Wonder's tasks aren't in it at all. Pinning control's models would remove the question, but provider usage limits make a pinned arm hard to keep running.
Some tasks escaped their arm. Six cheap-pool tasks ran a model outside the cheap pool, such as Muse Wonder, Sol or Sonnet. They are counted in the arm they were assigned to, as intention-to-treat requires. Ten more were left out because a crew had already been set on the task by hand.
What I take from it
On this repository's medium work, the cheaper models were not visibly worse on any settled quality measure, and they cost about half as much where cost could be priced. That is enough to keep testing, and not enough to switch. The delivery dip and the merge-conflict rise are small and uncertain. The regression data isn't in yet. The priced subset leaves out every Muse task, and the control arm wasn't a fixed baseline.
The next experiment applies the same question one tier up, with an arm per model on hard tasks, so a pass or fail points at a specific model.
Scope: one repository, Orbit's own, over three days, at about 115 tasks per arm. It is not a benchmark of the models. The per-arm readout as Orbit recorded it at conclusion is in readout.json, and method.md describes how it was computed.