Agent benchmarks carry a double confound, and fixing it turns a flat leaderboard into a spectrum.

arXiv 2609.09218 names two ways a score fails to measure the model: execution-critical decisions get made by a fixed scaffold instead of the model, and the scorer grades output shape instead of task correctness. The repair protocol moves execution decisions to the model, swaps shape-based scoring for seeded ground truth, and reports worst-case and tail-risk metrics alongside the mean. On ComtradeBench the joint intervention converts a nearly flat leaderboard into a reliability spectrum. Auditing existing benchmarks, they found scorer validity is benchmark-specific while scaffold ownership was an uncontrolled axis everywhere they looked. Read it next to Real-SWE, which makes the opposite trade on purpose by keeping each model in its own harness.