Only 22 of 113 DeepSWE tasks predict full-benchmark scores

DeltaSelect resamples DeepSWE's published trials and finds that only 22 of its 113 tasks track full-benchmark results closely enough to trust. The paper's bar is a fifth-percentile Pearson correlation of at least 0.50, a threshold just 19.5% of tasks clear. The other 91 tasks don't reliably tell you whether a candidate change to a model or prompt helped.

DeltaSelect also works as a method. It picks the subset of tasks whose single-run results predict the full benchmark, then fits that fixed set to a dollar budget. Applied in a case study on gpt-5.6-luna, it drove skill and instruction revisions across 13 evaluations for $27.86 total.

The eval that came out the other side ran 58.1% cheaper per run, $1.75 against $4.18 (p=0.008), and scored higher on the calibrated metric, 42.36% against 36.46%. Teams running the full 113-task suite on every candidate change are paying for 80% of that spend on tasks the paper says carry no signal. The fix is picking the fifth of the suite that predicts the whole benchmark, not adding more tasks to it.