SWE-NFI grades AI refactors on structure, not tests alone

SWE-NFI exposes a split in how AI coding agents perform: they pass functional tests but barely improve code structure, per the study behind it. The benchmark pulls 188 tasks from merged Python pull requests and grades each agent submission against 92 executable rules. That setup separates two questions most coding benchmarks blur together: did the tests still pass, and did the code actually get better.

That distinction matters for anyone routing refactoring or performance tickets to an agent. The best-performing agent in the study cleared 70.0% of functional-correctness tasks but scored only 0.0 to 1.3 on structural improvement, against a 1.5 human reference. Every agent the researchers evaluated followed the same pattern: strong on correctness, weak on structure, with none coming close to the human baseline.

A green test suite is not evidence a refactor made the code better. It proves the agent didn't break anything a test checks for, and nothing more. I'd treat that as the working rule: let agents run on behavior-changing work where tests can catch regressions. But put a human on the diff for anything labeled refactor or performance, because the tests won't tell you if the structure improved.

The gap is worth watching as more teams lean on agents for maintenance work instead of only new features. If a future version of this benchmark shows agents closing the structural-improvement gap without a corresponding drop in correctness, that changes the calculus. The 92 rules behind SWE-NFI are already doing work that code review used to do alone.