BM25 Outscores Dense Retrieval Past 10M Tokens

A corpus-scaling benchmark released July 30 shows BM25 beating dense retrieval by 20 points past 10 million tokens, per USTC and Metastone Technology researchers.

Most enterprise RAG systems ship with dense embeddings as the default retriever. I've built two that way myself. At the benchmark's full scale, 601 million tokens across 511,959 documents, BM25 scored 50.5 against dense retrieval's 29.9. The gap widens the bigger the corpus gets.

The design is why this result holds up where others don't. The benchmark, called EnterpriseRAG-Bench, nests 28 tiers across roughly a 450x range, each 1.25x larger than the last. The relevant documents and distractors stay identical at every tier. Only the background corpus grows, so corpus size is the one thing moving, not the queries or the judgments.

The mechanism is uncomfortable once you see it. Embeddings compress meaning into a fixed number of dimensions, and every document you add crowds that space further. Exact term matching doesn't degrade the same way. A rare part number or ticket ID stays rare no matter how many documents pile up around it.

One caveat: this is one benchmark on enterprise-style documents, dense with proper nouns and IDs. That's exactly where lexical search has the edge. The paper doesn't test conversational or narrative corpora, where embeddings might hold up better.

Any team running pure dense retrieval past 10 million corpus tokens has a measurable accuracy problem, not a theoretical one. The fix is BM25 as a first-class scorer, combined through reciprocal rank fusion, and that's closer to a weekend patch than a rebuild.

A Thoughtworks measurement published the same day points at the same lesson from a different angle. Fifteen refactoring steps on a 17,155-line Rust file cut input tokens per change 83%, from 159,564 to 27,360. The savings came from shrinking what an agent had to read. How you organize what gets retrieved matters more than which model reads it.