Self-improving agent harnesses don't beat simple sampling, paper finds

Evolved agent harnesses lose to simple test-time scaling once you match compute budgets, per a paper from Wang, Hajishirzi, Tsvetkov, and Dasigi. That's the control experiment self-improving agent loops keep skipping. On Terminal-Bench 2.1, harness evolution didn't consistently beat plain sampling, and the gains didn't carry to held-out tasks.

The paper, arXiv 2607.12227, flags two holes in how harness evolution gets graded. Nobody compares it to a simple baseline at matched compute. And everyone reports scores on the same public benchmark the search used to pick winners.

The fix is boring. No new tricks. Take the token budget your evolution loop spends searching harness variants. Spend the same budget on N parallel samples with a picker instead. If the evolved harness doesn't beat that baseline, you've tuned an overfit, not built a better harness.

I run agent loops that rewrite their own prompts after each cycle based on what worked. This is the control experiment I haven't run: a held-out benchmark my evolution step never sees, scored against a matched-budget baseline. The paper doesn't say if this holds past Terminal-Bench 2.1, but that's the right amount of claim for what they tested.