Speculative Decoding Pays Off More at Heavier Quants

Speculative decoding speeds up heavier-precision quants more than lighter ones, a sweep of ten configs on Qwen3.6-27B found, per a post on r/LocalLLaMA. Local model builders default to the lightest quant they can run for speed. This test says that instinct works against you once a draft model enters the picture.

The sweep ran ten speculative-decoding setups across multiple quantization levels and found the same ranking held every time: the heavier the quant, the bigger the speedup from the draft model. That's a monotonic result, unusually consistent for a single hobbyist test.

The common wisdom runs the other way. Low-bit quants are the ones assumed to be memory-bound, and memory-bound decoding is exactly what speculative decoding is supposed to fix, so the smallest models were expected to gain the most. This benchmark flips that.

It's one Reddit thread, self-reported, with no independent replication. The methodology is stated and the ranking holds across all ten configs, which is more rigor than most hobbyist benchmarks show, but it's still a single source.

If the ranking holds up outside this one thread, running the smallest quant for speed is the wrong call once speculative decoding is in the mix. Worth watching whether anyone reruns this on a different model family before calling it settled.