Per-chunk RAG verification is worse than no filtering at all on multi-hop questions — and gets worse as your generator gets better

arXiv 2608.00585·high signalXBlueskyLinkedInCopy link

Scoring each retrieved chunk and dropping the failures assumes one chunk is a sufficient premise for the answer; multi-hop questions are constructed so that none is. Entailment scoring reaches only 0.643 / 0.523 / 0.560 AUC on HotpotQA, 2WikiMultihopQA and MuSiQue against 0.951 on single-hop SQuAD, and end-to-end across three datasets, three generator sizes and two prompts, per-chunk gating was significantly worse than not filtering in every single cell — with the penalty growing as the generator gets stronger. The repair is cheap and available: condition verification on the decomposed sub-question rather than the original query, which lifts later-hop entailment from 0.546 (chance) to 0.840 with gold decompositions, and an off-the-shelf Qwen2.5-7B decomposer captures 31% of that ceiling. Iterative retrieval systems already generate these decompositions and throw them away before verifying.