Pooled LLM judging preserves 97% of system orderings at up to 4.9x lower cost.

This method has an LLM judge the union of documents retrieved by the current candidate set, then expand the pool incrementally by judging only new documents each system contributes, reusing judgments across systems. Validated on four retrieval benchmarks with 11 systems and deployed to compare 62 retrieval configurations for a financial news QA system, pooled rankings correlated strongly with gold-standard evaluation, 97% of pairwise orderings survived bootstrap uncertainty, and production document overlap gave 65-80% judgment reuse.