Pick your reranker by latency budget: FlashRank (15–30ms CPU) vs cross-encoder (100–250ms) vs Cohere API (150–400ms)

Vaibhav Dixit·medium signal

The reranking stage has a concrete latency/quality tradeoff worth engineering against: a MiniLM cross-encoder over N=50 candidates runs 100–250ms on CPU, FlashRank drops that to 15–30ms, and Cohere's hosted Rerank adds 150–400ms plus network. Merge the hybrid result sets with Reciprocal Rank Fusion (RRF) before reranking so dense and keyword hits are normalized into one ordering.