Cross-Encoder Re-ranking After Initial Retrieval: 18-42% Precision Gain with Net Token Cost Savings

Abhishek Gautam·medium signal

Adding a cross-encoder re-ranking stage after vector retrieval — where each (query, chunk) pair is scored independently rather than using embedding similarity — improves retrieval precision by 18-42% in production evaluations. The 50-200ms per-batch latency overhead is offset by passing fewer, higher-quality chunks to the LLM, reducing downstream generation token costs. At scale, LLM savings routinely exceed re-ranker compute cost, making this a net-positive optimization. Best cross-encoders in 2026: Cohere Rerank v3.5, Jina Reranker v2, and the open-source BGE-Reranker-v2-m3.