TreeWY Removes Per-Node State Snapshots From Speculative Decoding on Gated DeltaNet Hybrids

arXiv 2608.20961·medium signalXBlueskyLinkedInCopy link

TreeWY (arXiv 2608.20961, Aug 21) attacks a specific cost in serving hybrid models where most layers are linear-attention Gated DeltaNet carrying a fixed-size recurrent state: to verify and roll back draft tokens, current systems snapshot the full recurrent state at every draft position, and those snapshots cannot be shared across draft-tree branches. Using a tree-structured WY transform of the gated delta rule, it computes every draft node's output with a single triangular solve and reconstructs only the accepted state on commit, storing a small pseudo-value matrix instead. On Qwen3.5 35B and 397B serving benchmarks this cuts speculative state memory and KV-cache pressure at identical acceptance length, raising throughput and lowering time-to-first-token where memory binds, at a few percent cost where it does not.