Race-free async scheduling and a FlashInfer GDN backend push vLLM to 25K tok/s/GPU — with the exact serving flags published

vLLM Blog·medium signalXBlueskyLinkedInCopy link

vLLM documented reaching 25K total TPS/GPU on Qwen3.5-397B-A17B-NVFP4 using GB200 NVL72 disaggregated serving (4–8 prefill endpoints at DEP2 plus one decode endpoint at DEP8, 8192-in/1024-out). Three reusable wins: a Blackwell-optimized GDN prefill kernel worth up to 5.92x kernel and 1.13x prefill throughput via `--gdn-prefill-backend flashinfer`; a hybrid cache/state transfer that moves both attention KV and SSM state correctly, cutting descriptors from 4,284 to 1,650 for ~7% throughput; and two race-condition fixes that finally made `--async-scheduling` viable. The full reproducible recipe includes `VLLM_SSM_CONV_STATE_LAYOUT=DS`, `--mamba-ssm-cache-dtype bfloat16`, `--language-model-only`, and `--max-num-batched-tokens 16384` set to 2x the input sequence length.