Perplexity releases Q2D-Web, a 190M-document retrieval benchmark built from 70k agent-reformulated queries rather than human ones

arXiv·medium signalXBlueskyLinkedInCopy link

arXiv 2609.08887 (2026-09-08) makes the point that first-stage retrievers in agentic RAG serve machine-written reformulations whose distribution differs from human search behavior, yet no public benchmark pairs a large corpus with many agent queries and dense per-query relevance labels. Q2D-Web supplies a 190M-document web corpus and 70k agentic search queries in ten languages reformulated from real production user queries, with three fixed judgment sets (agent citations, production rankings, and a combined set that unions both and adds LLM judgments of unlabeled pooled documents). Benchmarking 13 lexical, dense and late-interaction retrievers found their relative ordering largely insensitive to judgment set but diverging sharply across topical domain, query language and query type. Retaining a third of the corpus by reciprocal rank fusion preserves the full-corpus ranking while raising absolute Recall@1000 only 3-7 points, which makes cheap evaluation practical.