The best code-embedding retriever returns a functionally broken near-clone as rank 1 in two thirds of queries

arXiv 2609.01865·high signalXBlueskyLinkedInCopy link

ExecRetrieval (arXiv 2609.01865, 2026-09-01) builds a search pool that contains the counterfactuals: 939 Python tasks, each with one execution-verified canonical implementation and up to four execution-verified buggy variants produced by a single targeted mutation. Across 23 dense embedding configurations plus BM25 under provider-native invocation, the top hosted system reaches exec@10 = 1.00 but only exec@1 = 0.331, and rank-1 misses are paired buggy variants 91.5-99.4% of the time on the four leading systems. The canonical scores below at least one of its four paired distractors in 67-78% of queries, so any coding agent that trusts a single top-1 embedding hit is being handed subtly wrong code most of the time.