Reliable, Developer-Aligned Evaluation of SE Agents: 279-Paper Review Finds an 'Illusion of Competence'

arXiv·low signal

An FSE '26 paper argues current coding-agent benchmarks (SWE-bench, SWT-bench, AgentBench) capture only narrow, synthetic slices of behavior and create an 'illusion of model competence.' From a systematic review of 279 peer-reviewed papers across 26 coding tasks, the authors propose contamination-aware, trajectory-aware, in-the-wild evaluation — using coding agents' tell-tale commit signatures (Codex, Copilot, Claude Code, Google Jules) to study real agentic vs human contributions and their longitudinal impact on maintainability. A useful counterweight for anyone taking leaderboard SWE-bench numbers at face value.