Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-15 · high
The paper wants agents graded by real commit history over time, not one-shot synthetic scores like SWE-bench.
Why now: The paper landed in the July 15 briefing alongside a separate METR finding reaching the same conclusion about benchmark trust.
A review of 279 papers finds AI coding benchmarks measure a narrow synthetic slice of work, not real skill, per the arXiv preprint.
That's a warning for anyone choosing a coding agent by leaderboard score alone. A benchmark result is being treated like a hiring signal for a system that's never touched a real codebase.
The paper, posted ahead of FSE '26, singles out SWE-bench, SWT-bench, and AgentBench by name. It calls the gap between benchmark score and real skill an "illusion of competence."
Instead, the authors propose contamination-aware, trajectory-aware evaluation done in the wild. That means tracking an agent's actual commit signatures over time, comparing the record against what a human engineer would have produced. That's harder to game than passing a fixed synthetic test suite.
A related METR finding on something called Sol reaches the same conclusion from a different angle. That adds to the case against taking leaderboard numbers at face value. Swap in contamination-aware, in-the-wild testing for one-shot scoring, and this paper's bet is that the current agent leaderboard order doesn't survive.
Each link below shares sources, entities, or timing with this story.
Shared entity: SWE / Same source domain / Shared topic / Earlier coverage
Both cover SWE; reported by the same outlet (arxiv.org); overlapping topics (agent, argu, benchmark, current).
Both cover SWE; reported by the same outlet (arxiv.org); overlapping topics (agent, benchmark, paper).
Both cover SWE; reported by the same outlet (arxiv.org); overlapping topics (agent, benchmark, commit).
Shared entity: SWE / Shared topic / Earlier coverage / Tension
Both cover SWE; overlapping topics (agent, benchmark, current); earlier SWE coverage from 2026-02-12.
Shared entity: SWE / Same source domain / Shared topic / Earlier coverage
Both cover SWE; reported by the same outlet (arxiv.org); overlapping topics (agent, benchmark).
Both cover SWE; reported by the same outlet (arxiv.org); overlapping topics (agent, commit).
Shared entity: SWE / Shared topic / Earlier coverage
Both cover SWE; overlapping topics (agent, benchmark, evaluation, paper); earlier SWE coverage from 2026-04-12.
Shared entity: SWE / Same source domain / Shared topic / Earlier coverage
Both cover SWE; reported by the same outlet (arxiv.org); overlapping topics (agent, commit).