Sliding-window fix cuts speculative decoding cost 28-44%

Alagappan Valliappan's new paper found the built-in draft step in speculative decoding scans the full cache every step, becoming the costliest part at million-token context. At that length, the draft step can end up costing more than it saves, according to the paper. That matters for any inference stack running near a million tokens, since the fix recovers 28-44% of the wasted cost.

The fix borrows a StreamingLLM-style trick: a sliding window plus an attention sink, applied only to the draft's attention. That drops about 99% of the draft's KV cache entries at 1M-token context. It was tested across Qwen GDN-MoE at 35B and 122B parameters and a Mamba2-hybrid NoPE model at 120B, running in SGLang on a single GPU. That combination cut per-decode-step cost 28-44%.

It's lossless, not an approximation: the full-attention target model still verifies every token the draft proposes, so accuracy doesn't move. The KV entries dropped from the draft's window aren't wasted either. A ring buffer reclaims the 7.7-11% that never get read, at no cost to acceptance rate.

The bigger point here is that a step marketed as negligibly cheap turned out to dominate cost once context got long enough. That's worth remembering the next time an inference paper calls something free.

It's a fix teams can apply without retraining, which matters as more inference stacks push into million-token context.