Sequential Monte Carlo Speculative Decoding: 5.2x Speedup Over Autoregressive, 2.36x Over Standard Spec Decoding

arXiv·high signal

SMC-SD from Cornell/MIT/ETH replaces token-level rejection sampling with importance-weighted resampling over draft particles. Because LLM inference is memory-bandwidth-bound, the arithmetic to draft and score particles in parallel comes nearly free—no rollback needed. Achieves 5.2x over autoregressive and 2.36x over standard speculative decoding while staying within 3% accuracy on reasoning, instruction-following, and coding benchmarks.