A Router That Predicts Output Length and KV Pressure Edges Out Least-Loaded on Disaggregated Serving

arXiv 2609.16206·low signalXBlueskyLinkedInCopy link

Calibrate, Then Route (arXiv 2609.16206, submitted 14 Sep 2026) studies request routing for disaggregated LLM serving, where prefill and decode already sit on separate GPU pools under systems like DistServe, Splitwise, and Mooncake but routing still decides which instance takes each request. Their router estimates additional completion time from exact prompt length, predicted output length, post-admission KV cache pressure, and SLO class, validated on eight NVIDIA A40 GPUs running vLLM with NIXL moving KV caches between pools. Across three bursty traces at measured saturation it reached 0.864 mean goodput versus 0.835 to 0.847 for round robin, least loaded, and a length heuristic, with the lowest variance, and the authors note hardware calibration matters since simulator-derived constants cost part of the gain.