Pooled LLM Judging Preserves 97% of System Orderings at Up to 4.9x Lower Evaluation Cost

arXiv 2609.02745·medium signalXBlueskyLinkedInCopy link

Choosing a retrieval model for production RAG needs repeatable comparison, but relevance judgments are expensive and have to be redone as candidates arrive. This work has an LLM judge the union of documents retrieved by the current candidate set, then expand the pool incrementally by judging only the new documents each new system contributes, reusing all judgments across systems. Validated on four retrieval benchmarks with 11 dense, sparse and hybrid systems and deployed to compare 62 retrieval configurations for a financial news QA system, pooled rankings correlated strongly with gold-standard evaluation, 97% of pairwise orderings survived bootstrap uncertainty in the qrels, and production document overlap gave 65-80% judgment reuse.