Infra2026-06-23 · source-backed
vLLM ships multi-tier KV-cache offloading and batch-invariant FP8 for a 28.9% latency cut.
Story
June 2026 releases add a multi-tier KV-cache offloading framework (Python filesystem tier plus Mooncake disk offload, extending past CPU memory), make Model Runner V2 the default for Qwen3 dense models, and land Cutlass FP8 for batch-invariant inference. (vLLM) If you self-host inference, the KV offloading tiers let you serve longer contexts and higher concurrency on the same GPUs. The batch-invariant FP8 also gives reproducible outputs across batch sizes, which quietly fixes a real headache for evals and caching where the same input shouldn't drift by batch.
Related stories
Each link below shares sources, entities, or timing with this story.
Shared entities / Same source domain / Shared topic / Earlier coverage / Downstream implication
llama.cpp Gets Backend-Agnostic Tensor Parallelism, and the NVIDIA Tax Just Got Optional
Both cover CPU, GPUs, Qwen3; reported by the same outlet (github.com); overlapping topics (context, inference, model).
Shared entity: FP8 / Same source / Shared topic / Earlier coverage
vLLM v0.20.2 Ships Model Runner V2: 56% Throughput Improvement on GB200.
Both cover FP8; cite the same source (vLLM); overlapping topics (inference, model).
Shared entity: Qwen3 / Same source domain / Shared topic / Earlier coverage
Gemma 4 Ships Under Apache 2.0 and the Numbers Are Absurd
Both cover Qwen3; reported by the same outlet (github.com); overlapping topics (dense, inference, model).
llama.cpp just made low-bit KV cache dramatically more accurate, nearly for free.
Both cover Qwen3; reported by the same outlet (github.com); overlapping topics (caching, context).
Shared entity: CPU / Same source domain / Shared topic / Earlier coverage
Microsoft Foundry-Local: One SDK to Replace Ollama, llama.cpp, and Whisper.cpp. No Cloud Required.
Both cover CPU; reported by the same outlet (github.com); overlapping topics (context, model).
Shared entity: Qwen3 / Same source domain / Shared topic / Earlier coverage
Nvidia's Blackwell Ultra Just Made Self-Hosted Inference a Real Option: 2.49M Tokens/Sec, 30 Cents Per Million
Both cover Qwen3; reported by the same outlet (github.com); overlapping topics (inference, model).
Shared entity: Python / Same source domain / Shared topic / Earlier coverage
Timber — Ollama for Classical ML (476 stars, 188 HN points)
Both cover Python; reported by the same outlet (github.com); overlapping topics (inference, model).
Shared entity: Qwen3 / Same source domain / Shared topic / Earlier coverage
Alibaba Launches Qwen 3.5 — The First Agentic-Native Foundation Model
Both cover Qwen3; reported by the same outlet (github.com); overlapping topics (context, model).
Source trail
Entities
Provenance
- Canonical issue
- Ramsay Research Agent — June 23, 2026
- AI generated
- no
- Story unit
- 2026-06-23-vllm-ships-multi-tier-kv-cache-offloading-and-batch-invariant-fp8-for-a-28-9-latency-cut
- Labels
- source-backed, canonical briefing excerpt