Chunk Coverage scores RAG retrieval like code coverage

A new metric called Chunk Coverage counts how many corpus chunks a RAG test suite actually retrieves, according to a paper posted at arXiv 2607.18155.

Most RAG test suites grade only the final answer and leave the retriever itself untested, per the paper. On clinical and financial scenarios, coverage-guided test selection reached 50% coverage 1.7 times faster than random picks.

Coverage-guided selection also beat picks biased toward redundant, already-tested chunks, hitting 50% coverage 4.2 times faster than that approach. Fault detection, measured by APFD, improved 10 to 25% over random selection.

What to check

Retrieval failures are quiet. A wrong or missing chunk still produces a fluent, confident answer, so teams rarely notice the gap without a way to measure it. Chunk Coverage gives them a number to track after every corpus update, and a way to choose which test queries actually stretch the retriever.

The paper doesn't say how these gains hold up outside clinical and financial scenarios, so treat the multipliers as domain-specific until someone tests a third.