vToken Reclaims 27-72% of Wasted KV Cache Blocks in vLLM Because PagedAttention's Block Granularity Fights Token-Level Eviction

arXiv 2608.13263·high signalXBlueskyLinkedInCopy link

vToken (arXiv 2608.13263, submitted 2026-08-13) identifies a mismatch nobody had quantified: KV eviction algorithms like H2O and Scissorhands operate at token granularity while PagedAttention manages fixed-size blocks, so evicted tokens leave intra-block holes that are never reclaimed. Their token-table indirection layer with asynchronous live-token repacking preserves PagedAttention kernels and CUDA Graph compatibility, cutting retained KV blocks per request by 27.2%-72.3% and improving SLA-constrained throughput up to 1.37x. Under a constrained active-KV budget it doubles maximum feasible concurrency, and shrinks per-eviction-policy integration from 500+ lines to under 50 — a real vLLM patch, not a simulation.