Same model, nine harnesses, 17.5x spread in cost per task

Runta published FrontierHarness on September 2 and it's the most directly useful benchmark I've read this quarter, because it controls the one variable everyone conflates.

Nine agent harnesses (Codex, Claude Code, OpenCode, Pi, Oh My Pi, DeepSeek Harness, Kimi Code, Exo Harness, Hermes) on software engineering and terminal tasks. Every run starts cold with a fresh checkpoint restore, which kills the warm-cache advantage that makes most harness comparisons meaningless. Cost per task ranged from $1.05 to $18.34. That's 17.5x. Cost per passing task, which is the number you should actually budget against, ranged $0.0615 for OpenCode to $0.2880 for Claude Code, about 4.7x. Codex led on quality at 66.7% pass rate for $3.47 per task. Exo Harness was cheapest at $1.05 with 53.3% success.

Read those two spreads together. The cost-per-task gap is 17.5x and the cost-per-passing-task gap is 4.7x, meaning the expensive harnesses are buying real success rate with their spend, but nowhere near proportionally. You pay 17x more and get roughly 4x less waste. That's a bad trade if you're running volume and a fine trade if every failed task costs you a human hour of cleanup.

The methodology point deserves its own paragraph. Cold start, fresh checkpoint, every run. Most harness benchmarks I've seen let prompt caches warm up across runs, which flatters whichever harness has the stickiest cache strategy and tells you nothing about your Monday-morning first task. Killing that bias is why these numbers are usable.

Here's what I'd do with it. Stop treating harness choice as an ergonomics decision. I've picked coding agents based on how the diff view feels, and I'd guess most people reading this have too. That was defensible when the spread was 20%. At 17.5x on the raw number it's a budget line item, and if you run agents in a loop against a benchmark or a test suite, it's the largest budget line item you have. Measure your own workload across two harnesses for a week before you standardize on one.

The result also reframes every model-pricing argument from the past month. Anthropic cut Fable 5.1 cache reads 75% to $0.25/Mtok. Real money. But a 75% cut on one input class inside a harness that costs 17x more than the alternative is noise against the harness decision. The model price list is the thing everyone watches and the harness is the thing that actually sets the bill.

One caveat I'd want closed: FrontierHarness doesn't publish, as far as I can tell from the report, how much of the cost spread comes from context-management strategy versus tool-call chattiness versus retry behavior. Those are three different fixable problems and knowing which one dominates would turn this from a shopping guide into an engineering brief.