Public story · 2026-07-02 · high
RepoRescue benchmarks agents on whole-repo dependency fixes
It's a harder, more realistic test than the single-function fixes most benchmarks use for judging legacy-code agents.
Why now: As of July 2, most cited agent benchmarks still score single-function fixes, not the multi-file breaks RepoRescue tests.
Story
RepoRescue tests coding agents by breaking a codebase with dependency and version changes spread across many files at once, per the paper posted on arXiv. That's harder than the single-function fixes most benchmarks score. It's also the test that predicts whether an agent can be trusted on an old codebase, not a fresh one.
Most published agent benchmarks still score single-function fixes: patch one broken call, pass the test, done. RepoRescue skips that setup. It drops agents into a repo where a dependency bump or version change has broken things across many files at once. That's closer to what happens when a legacy project falls behind on updates.
Anyone who's tried to push a coding agent through a multi-file version bump already suspects single-function benchmarks hide the hard part.
Single-function benchmarks are the wrong yardstick for judging whether an agent can handle real legacy work. The model that handles RepoRescue's whole-repo breaks best is the one worth trusting with an actual dependency migration.
As of July 2, most cited agent benchmarks still score single-function fixes, not the multi-file breaks RepoRescue tests.
Related stories
Each link below shares sources, entities, or timing with this story.
Shared entity: Anyone / Same source domain / Shared topic / Earlier coverage / Tension
The Red Queen Gödel Machine co-evolves agents with their own evaluators.
Both cover Anyone; reported by the same outlet (arxiv.org); overlapping topics (agent, anyone, benchmark).
Shared entity: Anyone / Shared topic / Earlier coverage
Claude Agent SDK and `claude -p` now draw from a separate monthly credit.
Both cover Anyone; overlapping topics (agent, anyone, chang); earlier Anyone coverage from 2026-06-17.
Google's Agent Development Kit Had an Unpinned LiteLLM Dependency During an Active Supply Chain Compromise
Both cover Anyone; overlapping topics (agent, dependency, version); earlier Anyone coverage from 2026-03-30.
Same source domain / Shared topic / Downstream implication
Frontier model "manipulation" is task-dependent, not a single trait.
Reported by the same outlet (arxiv.org); overlapping topics (agent, anyone, deploying); traces where this leads (implication).
Same source domain / Shared topic / Tension
Top coding agents secretly write Python to write your obscure language for you.
Reported by the same outlet (arxiv.org); overlapping topics (agent, better, coding); pushes against this story (but).
Engram: a lean retrieved context beats stuffing the full history into the window.
Reported by the same outlet (arxiv.org); overlapping topics (agent, benchmark, better); pushes against this story (versus).
"Strained coherence" might be the early-warning metric for agents going off the rails.
Reported by the same outlet (arxiv.org); overlapping topics (agent, anyone, coding); pushes against this story (against).
Your AI-Generated Code Compiles. It Doesn't Work.
Reported by the same outlet (arxiv.org); overlapping topics (agent, broken, coding); pushes against this story (but).
Source trail
Entities
Claim evidence
- RepoRescue benchmarks agents on whole-repo dependency fixes
Provenance
- Canonical issue
- 2026-07-02
- AI generated
- yes
- Story unit
- 2026-07-02-reporescue-benchmarks-agents-on-whole-repo-compatibility-rescue
- Labels
- source-backed, canonical briefing excerpt