Tools
vLLM cuts six identical token-mapping kernel launches per step down to one
PR #57102 (merged 2026-09-18 03:14 UTC) found that `build_attn_metadata()` launched one token-to-request mapping kernel per KV cache group even though every group in a call sees identical query boundaries. A DeepSeek V4.1 spec-decode trace showed 6 launches in target prep and 3 in draft prep per step, all computing the same thing. A three-line per-call cache took target launches 6→1 and draft 3→1, total kernel launches 30→25 and 12→10, and cut host wall time per call by 11.1% (target) and 12.3% (draft) on GB200.
Source
↳ Follow the thread