llama.cpp Moves Sampling Off the CPU for Speculative Decoding — Practitioners Report 8% More tok/s on a 5090
r/LocalLLaMA (131 upvotes) flagged the in-flight llama.cpp work moving sampling from CPU to GPU for MTP-enabled setups, reporting an 8% tokens/sec gain on an RTX 5090 with Qwen3.6:35B and independent confirmation from a P40 owner. The corresponding PR is #25532 'Backend sampling multi output' by gaugarg-nv (NVIDIA), open since July 10 and still active as of 11:39 UTC today: +1,127/-340 across 21 files, extending backend sampling from one output per sequence to multiple outputs so it works during speculative decoding, and making distribution sampling deterministic between CPU and backend paths. If you run speculative decoding locally, this is a free single-digit-percent throughput win once merged — and the determinism guarantee matters more than the 8% for anyone benchmarking.
↳ Follow the thread