Harness optimization is now a measurable model capability — and the optimizer model matters more than the coding harness it runs through

arXiv 2608.06301·medium signalXBlueskyLinkedInCopy link

HarnessOpt-Bench has a frontier LLM act as an optimizer that receives a target agent's seed harness (prompts, tools, control flow, memory, orchestration code) plus graded eval feedback and a fixed target-evaluation budget, then edits the harness and nominates a candidate scored by normalized gain on a held-out partition inaccessible during optimization. Across 5 frontier models, 4 downstream tasks, and 111 scored runs, optimizer models separated more than the coding harnesses they acted through, and native harnesses did not consistently beat shared ones. The practical read for builders: when you hand your agent scaffold to a model to improve, spend on the strongest optimizer model rather than standardizing the tooling around it — and always hold out a test partition, since self-graded harness edits overfit the feedback set.