Mirror Speculative Decoding: Heterogeneous GPU+NPU Parallel Draft-Verify for 2.8–5.8x Inference Speedup

ICLR 2026·medium signal

Mirror-SD (ICLR 2026) eliminates the synchronous bottleneck in standard speculative decoding by running draft-model speculation and target-model correction paths simultaneously on heterogeneous hardware—draft to NPU, target to GPU—removing resource contention while maintaining mathematical output equivalence to the original model. Speculative streaming emits multiple draft tokens per step while the target model processes the previous batch in parallel, and cross-device parallelism on commodity GPU+NPU setups is the key deployment primitive. Benchmarks show 2.8x–5.8x wall-time speedup on SpecBench across 14B–66B parameter models—a 30% relative improvement over the strongest EAGLE3 baseline—making this the current state-of-the-art for latency-sensitive production inference without quality degradation.