Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-16 · high
A token-table indirection layer repacks live tokens around the holes, lifting SLA-bound throughput up to 1.37x under tight KV budgets.
Why now: The arXiv paper posted on August 16 with a working vLLM patch attached, not just benchmark numbers.
vToken recovers 27.2% to 72.3% of the KV cache blocks vLLM wastes during eviction, per a paper posted August 16. That waste caps how many requests a server can hold in memory. Recovering it lifts SLA-constrained throughput as much as 1.37x and, under a tight active-KV budget, doubles the concurrency a server can handle.
The mismatch is architectural. Eviction algorithms like H2O and Scissorhands drop individual tokens, but PagedAttention only frees memory in fixed-size blocks. Evict a token from the middle of a block, and the hole sits unusable until the whole block empties.
vToken's fix is a token-table indirection layer that tracks which tokens in a block are still live. Asynchronous repacking then moves those tokens together to free whole blocks, without changing PagedAttention's kernels or breaking CUDA Graph compatibility.
The paper also reports that adding a new eviction policy now takes under 50 lines of code, down from more than 500. It calls vToken a working vLLM patch, not a simulation.
Each link below shares sources, entities, or timing with this story.
Shared entity: Under / Same source domain / Shared topic / Earlier coverage / Tension
Both cover Under; reported by the same outlet (arxiv.org); overlapping topics (budget, under).
Shared entity: Under / Same source domain / Shared topic / Earlier coverage
Both cover Under; reported by the same outlet (arxiv.org); overlapping topics (asynchronou, token).
Same source domain / Shared topic / Tension
Reported by the same outlet (arxiv.org); overlapping topics (algorithm, cache, eviction, token); pushes against this story (against).
Shared entity: Under / Same source domain / Earlier coverage / Tension
Both cover Under; reported by the same outlet (arxiv.org); earlier Under coverage from 2026-08-07.
Both cover Under; reported by the same outlet (arxiv.org); earlier Under coverage from 2026-08-05.
Shared entity: Under / Same source domain / Earlier coverage
Both cover Under; reported by the same outlet (arxiv.org); earlier Under coverage from 2026-08-14.
Both cover Under; reported by the same outlet (arxiv.org); earlier Under coverage from 2026-08-12.
Shared entity: Nobody / Same source domain / Earlier coverage
Both cover Nobody; reported by the same outlet (arxiv.org); earlier Nobody coverage from 2026-08-07.