Reranker-First RAG: 60-80% Latency Reduction via Counterintuitive Context Filtering

dasroot.net·high signal

Adding a cross-encoder reranker to a RAG pipeline counterintuitively reduces total latency: the reranker adds 100-200ms overhead but removes irrelevant context from the LLM prompt, saving 3,400-6,800ms in inference time. The correct retrieval pattern is retrieve top-50, rerank to top-5, then pass to LLM — not retrieve top-5 directly. Combined with multi-tier semantic caching (exact-match sub-millisecond, similarity-match slightly higher), this cuts redundant LLM calls by 30-45% and is now integrated natively in LangChain v1.8 and RAGFlow v3.2.