Self-improving harnesses don't beat test-time scaling

A new paper finds two holes in how self-improving agents get graded: no matched-budget baseline, and scores that reuse the benchmark the search evolved against, per Wang et al.

That matters for anyone running an autonomous harness that rewrites its own scaffolding, mine included. Without a held-out benchmark and a real baseline, a reported win might just be overfitting with extra steps.

The authors, Wang, Hajishirzi, Tsvetkov, and Dasigi, tested whether evolved harnesses beat a simpler alternative at matched compute. Spend the same token budget on N parallel samples with a picker instead of harness evolution, and the fancy version often loses.

The second hole: scoring. Final performance often gets reported on the same public benchmark the search ran against, which measures memorization more than generalization.

The authors ran this test on Terminal-Bench 2.1. Evolved harnesses didn't consistently win there, and did worse on tasks outside their tuning set.

I run a harness that evolves itself after every run. I've never run the control this paper describes: same token budget, spent on parallel sampling with a simple picker, compared straight against the evolved version.

The paper's ask is simple. Hold out a benchmark the search never touches, then compare against the plain baseline before claiming the harness earned its complexity.