Comments only help AI coding tools when they leak the answer

A new paper puts the core assumption behind code-retrieval tools to a real test, and it fails. The LiveCodeBench study found that neither comment frequency nor comment intent predicted whether a model solved a problem correctly. For anyone feeding a coding agent nearby commented code as context, this paper measures exactly how costly the wrong kind is.

Researchers prefilled weaker models with comment blocks pulled from a stronger model's passing solutions. Pass@1 rose 17.2% on average. Comments pulled from a failed solution gave no reliable gain, and comments written for a different problem dropped pass@1 by 20.8%.

The gain traces to solution transfer, not commenting. A weak model reading a correct approach that a strong model already solved gets a boost, even when that approach arrives wearing comment syntax.

Take away the correctness and the gain disappears. Take away the relevance and it turns into a 20.8% loss.

The authors then tried to reproduce that gain through prompting alone, across many models and prompt variants. Best recovery was 24% of the external-comment effect. Prompting can't manufacture content the model never had.

Most code sitting near a given function was written for a different problem than the one a model is asked to solve now. A RAG setup that retrieves semantically similar chunks and prepends them retrieves that mismatched case by design, at scale.

There's a version that works: a correct worked solution to a closely related problem, labeled as such.

CrossCoder tested a different fix. It retrieves from a project's actual dependency source instead of similar-looking snippets. That adds up to 6.3% pass@1 on RepoExec, DevEval and a new benchmark called VersionExec, and it holds up across dependency version changes.