Improving BM25 Code Retrieval Under Frozen Tokenization: Adaptive q-Log Odds as a Drop-In Fix

arXiv·low signal

In retrieval-augmented coding, failures often begin when the relevant file is absent from the retrieved context. Under frozen generic tokenization where BM25 was built by a system the practitioner doesn't control, code-specific tokens get split poorly. This paper introduces adaptive q-log odds scoring as a drop-in BM25 replacement that compensates for suboptimal tokenization without reindexing. Practical fix for any RAG-based coding assistant using BM25 with a search system you can't modify.