A new benchmark says code retrieval ranks the wrong thing

Most retrieval systems score each passage against the query and rank by relevance. A paper posted to arXiv on September 18 argues that's the wrong unit for coding agents: relevance is a property of a single passage, but whether an agent can actually make its next decision depends on the whole set of passages together, per arXiv 2609.20050.

The failure mode is specific. A ranker rewarded for relevance will happily fill its budget with five near-duplicate passages that all confirm the same fact, and leave out the second fact the agent needed. High relevance scores, incomplete answer.

To measure this, the paper introduces SERBench: 500 held-out agent states pulled from 45 repositories, where a retrieved set only gets credit if it covers every fact the underlying decision required. That's a binary, decision-level bar instead of a graded relevance score.

The paper's own method, MSS-Complement, targets that bar directly. It makes three semantic calls: propose a candidate set, check what's missing, then fill the gap, returning 4 to 8 intact source units within a 6,144-token budget. On SERBench it reaches a complete set for 73.0% of states at five items and 80.6% at eight items. A Qwen3 embedding model with reranking, evaluated on the same states, hits 61.4% and 72.4%.

That's roughly an 8 to 12 point gap in favor of building toward completeness over ranking by similarity, on the same held-out states, at the same set sizes. The paper doesn't say whether that gap holds on languages or repo sizes outside its 45-repository sample, or what the three extra semantic calls cost in latency next to a single-pass reranker. For anyone building retrieval into a coding agent, that's the real question before adopting this approach: is a completeness check worth the extra round trip.