Scale AI's HarnessOpt-Bench Measures LLMs at Optimizing Agent Harnesses — and Finds the Optimizer Model Matters 1.8x More Than the Harness It Edits

arXiv / HuggingFace Daily Papers (33 upvotes)·high signalXBlueskyLinkedInCopy link

HarnessOpt-Bench (arXiv 2608.06301, Scale AI) tests whether frontier models can improve an agent system rather than write code: deliberately untuned seed agents on OfficeQA, GAIA, BrowseComp-Plus, and Terminal-Bench, with held-out test scores, expensive stochastic evals, and hard caps on eval calls and case passes. Swapping the optimizer model moves gain by 0.142 on average while swapping the harness moves it 0.079, and gains rise monotonically from +0.03 to +0.49 on OfficeQA across five GPT releases. The behavioral finding is the actionable one: broad exploration correlates positively with gains (Spearman +0.34 to +0.88) while detailed trace inspection correlates negatively (-0.31 to -0.64), and case-pass budget binds far harder than eval-call budget (82% vs 4% median utilization).