vLLM ships multi-tier KV-cache offloading and batch-invariant FP8 for a 28.9% latency cut.

June 2026 releases add a multi-tier KV-cache offloading framework (Python filesystem tier plus Mooncake disk offload, extending past CPU memory), make Model Runner V2 the default for Qwen3 dense models, and land Cutlass FP8 for batch-invariant inference. (vLLM) If you self-host inference, the KV offloading tiers let you serve longer contexts and higher concurrency on the same GPUs. The batch-invariant FP8 also gives reproducible outputs across batch sizes, which quietly fixes a real headache for evals and caching where the same input shouldn't drift by batch.