Four Model Generations Later, SWE-bench Patch Quality Hasn't Improved — Only Resolve Rate Has

arXiv 2607.18462·medium signal

Repository-level benchmarks track resolved rate across model generations but ignore whether the patches got any better. This study ran four generations of Claude and DeepSeek models on SWE-bench Lite and measured non-functional quality via CodeQL, CodeScene, CPU execution time, and peak memory. Newer models resolved more instances, but on tasks solved by both old and new models there was no consistent improvement in any non-functional indicator — most CodeQL differences were negligible and lost statistical significance after multiple-comparison correction, while memory usage actually crept up slightly in later models.