'Bug Fix' Tells You Nothing: Every Pair of Agentic SE Benchmarks Separates on at Least Two Demand Axes
The SNC profile characterizes repository-level coding tasks on three axes (Spread, Novelty, Centrality) grounded in empirical software engineering, applied to five widely used benchmarks and 14,922 trajectories from two model families at three scales. Every pair of benchmarks is statistically separated on at least two axes despite sharing nominal labels, and the separations trace back to specific curation decisions. Success concentrates in the low-SNC region for every family and scale, but the behavioral signature differs by family: Claude succeeds by matching the gold solution's scope, with its parity share on files rising from 0.17 at the smallest scale to 0.54 at the largest, while Qwen succeeds by exceeding gold scope at every scale.
↳ Follow the thread