Coding agents solve just 51% of real dependency upgrades

A new benchmark called DEPBENCH scores AI coding agents on real dependency upgrades that hide breaking changes. The best agent configuration tested solved just 104 of 203 tasks, 51.2%. That gap sits in the maintenance side of software work, chasing breakage no changelog mentioned. Most coding benchmarks measure writing new code, not this kind of upgrade toil.

DEPBENCH pulls together 203 tasks across five package ecosystems and five language communities. Each one hides code-level breakage that the upgrade itself never communicated to maintainers, the kind of incompatibility that only surfaces once the new dependency runs. Results varied widely by agent harness, model, and ecosystem, the paper found.

Most coding-agent benchmarks test greenfield work: write a function, pass a test, close a ticket. DEPBENCH tests something else, the grind of bumping a version number, then chasing every place the API contract shifted underneath it. No changelog note flags any of it. SWE-bench-style benchmarks mostly skip this maintenance half of the job, and now there's a number attached to it.

The paper doesn't say why the other 99 tasks failed, whether agents gave up, produced code that compiled but broke tests, or misdiagnosed the cause. Until agents close that gap, a human still has to review the result before trusting an upgrade to ship unattended.