vLLM removes a ROCm copy that cost 0.339 ms per decode step on Kimi-K3 FP4, attributed from profiles rather than source

GitHub·low signalXBlueskyLinkedInCopy link

vLLM PR #58566 (merged 2026-09-25) moves a contiguous copy in rocm_unquantized_gemm_impl into the only two branches that use it. Before, shapes that passed the broad skinny-GEMM gate but matched neither branch paid for a copy that was then discarded. On Kimi-K3 FP4 across 8x MI355X, per-step copies dropped from 73 to 4, one per KDA layer, saving 0.339 ms per decode step, almost all of it kernel-launch overhead. The author notes two earlier guesses made from reading source were wrong, and only kernel-timeline fingerprinting found the cause.