Automatic harness evolution doesn't beat simple test-time scaling — match budgets and hold out a benchmark before you believe your own numbers

arXiv 2607.12227·high signal

arXiv 2607.12227 (submitted July 14, Wang et al., incl. Hajishirzi, Tsvetkov, Dasigi) revisits automatic harness evolution for LLM agents and finds two methodological holes: methods are never compared against simpler baselines at matched compute budgets, and they report final performance on the same public benchmark they searched against. On Terminal-Bench 2.1, evolved harnesses do not consistently beat plain test-time scaling and generalize poorly. If you run a self-improving agent loop, this is the control experiment you probably skipped — spend the same token budget on N parallel samples with a picker, and hold out a benchmark the search never sees, or your harness gains are overfitting.