New Claude and DeepSeek models solve more SWE-bench tasks, not better ones

A new paper measures something SWE-bench leaderboards don't: whether the patches themselves get better. The authors ran four generations of Claude and DeepSeek models on SWE-bench Lite and scored the output with CodeQL, CodeScene, CPU execution time, and peak memory, per arXiv 2607.18462.

Resolve rate climbed, generation over generation, the way it always does on these leaderboards. But narrow the comparison to instances solved by both an old and a new model, and the non-functional numbers go flat. Most CodeQL differences were small enough to disappear after correcting for multiple comparisons. Memory usage actually crept up in later generations instead of down.

That's the finding worth sitting with. Four generations of frontier coding models, and the code they write for problems they can already solve hasn't gotten leaner, safer, or faster. It's gotten more coverage, not more craft.

I don't write eval harnesses for a living, but I read enough of them to know most teams grading coding agents still stop at pass/fail. This paper is a specific argument against that: pass/fail tells you an agent cleared a bar, not that the code behind it is worth shipping. CodeQL findings, memory footprint, and execution time are cheap to add to any harness that already runs the tests.

The paper doesn't say why memory ticked upward, whether it's more defensive code, more imports, or something else in how newer models solve the same problems. That gap matters if you're deciding whether a model upgrade actually improves your codebase's non-functional health or just its pass rate.