Optimizer Model Upgrades Beat Harness Rewrites 1.8x

Upgrading the optimizer model beats rewriting the harness by 1.8x on held-out test scores, per Scale AI's new HarnessOpt-Bench.

That's a 0.142 average score move from swapping the optimizer model, versus 0.079 from swapping the harness. Check the optimizer first, before spending an eval budget on scaffolding rewrites.

The behavioral finding is stranger. Broad exploration of the solution space correlated positively with gains, Spearman +0.34 to +0.88. Reading agent traces closely to hunt for the one bad tool call correlated negatively, -0.31 to -0.64.

Case-pass budgets bound the work harder than eval-call budgets too: 82% median utilization against just 4%.

I've burned entire evenings reading traces line by line looking for the failure. This paper says that time cost more than it returned. I'm not fully sold, trace-reading is usually how I find the hypothesis worth testing.

A harness that's betting the opposite way

The same day, PrimeIntellect-ai/prime-agent picked up 2,655 GitHub stars. That's about 3x the next-fastest repo on GitHub Trending, and it landed at No. 6 on Product Hunt.

Its pitch is a Continual Harness that edits its own prompts, skills, and subagent specs mid-run, built on a Recursive Language Model architecture. Prime Intellect claims 95.5% Best@1 on ARC-AGI-3 with Opus 5, edging a 95.4% human-expert baseline.

A related paper, ADIAS, backs a narrower version of the same idea. It carries a persistent issue list across optimization rounds instead of re-diagnosing from scratch each round. That's worth a 25.2% average improvement, and removing it drops performance up to 40.7%.