LLM Serving Schedulers Assume Poisson Arrivals; A WAIT Extension Estimates Burstiness Online and Beats vLLM, ORCA, and Sarathi-Serve

arXiv 2608.06135·low signalXBlueskyLinkedInCopy link

This paper (2608.06135, submitted 2026-08-06) attacks an assumption baked into most LLM inference scheduling work — Poisson request arrivals at a constant rate — which does not match real bursty traffic. It adds a lightweight online estimator of request intensity from observed interarrival times to the state-of-the-art WAIT algorithm, requiring no prior knowledge of the traffic pattern. Evaluated on Markov Modulated Poisson Process synthetic workloads with diverse request types, it reports higher throughput than Sarathi-Serve, ORCA, and vLLM in the tested low arrival-rate-shift scenarios at comparable latency. Note the evidence is simulation-only with no quantified margins in the abstract, so treat it as a direction rather than a deployable win.