Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-25 · high
The fix trims only the draft model's attention window, requires no retraining, and leaves accuracy untouched.
Why now: It's a fix teams can apply without retraining, right as more inference stacks push into million-token context windows.
Alagappan Valliappan's new paper found the built-in draft step in speculative decoding scans the full cache every step, becoming the costliest part at million-token context. At that length, the draft step can end up costing more than it saves, according to the paper. That matters for any inference stack running near a million tokens, since the fix recovers 28-44% of the wasted cost.
The fix borrows a StreamingLLM-style trick: a sliding window plus an attention sink, applied only to the draft's attention. That drops about 99% of the draft's KV cache entries at 1M-token context. It was tested across Qwen GDN-MoE at 35B and 122B parameters and a Mamba2-hybrid NoPE model at 120B, running in SGLang on a single GPU. That combination cut per-decode-step cost 28-44%.
It's lossless, not an approximation: the full-attention target model still verifies every token the draft proposes, so accuracy doesn't move. The KV entries dropped from the draft's window aren't wasted either. A ring buffer reclaims the 7.7-11% that never get read, at no cost to acceptance rate.
The bigger point here is that a step marketed as negligibly cheap turned out to dominate cost once context got long enough. That's worth remembering the next time an inference paper calls something free.
It's a fix teams can apply without retraining, which matters as more inference stacks push into million-token context.
Each link below shares sources, entities, or timing with this story.
Shared entities / Earlier coverage
Both cover Drop, MTP, Multi, SGLang; earlier Drop coverage from 2026-05-06.
Shared entities / Shared topic / Earlier coverage / Tension
Both cover GPU, SGLang; overlapping topics (decoding, draft); earlier GPU coverage from 2026-06-24.
SGLang competes with vLLM / Shared topic
Linked by a graph relationship (SGLang competes with vLLM); overlapping topics (context, cost).
Shared entities / Earlier coverage
Both cover GPU, MoE, MTP; earlier GPU coverage from 2026-07-15.
Both cover GPU, MoE, SGLang; earlier GPU coverage from 2026-06-04.
Both cover GPU, MoE, Multi; earlier GPU coverage from 2026-05-22.
Both cover GPU, MoE, SGLang; earlier GPU coverage from 2026-04-09.
Shared entities / Same source domain / Earlier coverage
Both cover GPU, MoE; reported by the same outlet (arxiv.org); earlier GPU coverage from 2026-07-23.