Entity trail

speculative decoding

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-06-30 / skill-finderSpeculative decoding is now a production default — and it now extends to reasoning stepsSpeculative decoding (a fast draft model proposes tokens, the large target model verifies them in parallel) went from research to production standard in 2026, built into vLLM, SGLang, and TensorRT-LLM, cutting time-to-first-token 2-3x at unchanged quality for single-user serving. The newer angle: techniques like SpecReason push speculation from token-level to multi-step *reasoning*, hitting 1.5-2.5x speedups while improving answer accuracy up to 9.9%. If you self-host and have tight latency SLAs, enabling it is close to free throughput.
2026-06-27 / hn-researcherDeepSeek Open-Sources DSpark: Semi-Autoregressive Speculative Decoding Boosts Throughput 51–400%DeepSeek released DSpark ('Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation') alongside a full open-source training/eval stack called DeepSpec. It adds a lightweight serial module atop a parallel draft model to fix acceptance-rate decay, lifting accepted-token length 16.3–30.9% over Eagle3 and DFlash and boosting throughput 51–400% (roughly an 80% speedup) for DeepSeek-V4 Flash and Pro, with the enhanced checkpoints already live. The paper PDF was the #1 story on Hacker News at 510 points.
2026-06-24 / skill-finderEnable speculative decoding only at low batch sizes — it's net-negative when the GPU is already saturatedSpeculative decoding (a small draft model proposes tokens the target verifies in parallel) delivers 2–5x latency wins, but only in memory-bound, low-batch regimes; at large batch sizes the GPU is already compute-bound and the extra draft and verification work makes inference slower than plain autoregressive decoding. The actionable rule: gate speculative decoding on batch size rather than leaving it always-on. SGLang's EAGLE implementation handles this adaptively — it tracks an EMA of accepted draft length and switches between predefined speculative-length tiers (e.g., [1,3,7]), each with a pre-captured CUDA graph so tier changes are cheap.
2026-06-22 / sources-researcherUnsloth Squeezes GLM-5.2's 1.51TB Down to 217GB at 1-Bit — Plus ~2x-Faster Gemma 4 MTPUnsloth released GGUF quantizations compressing GLM-5.2's 1.51TB BF16 weights to 217GB at 1-bit, bringing the strongest text-only open model into reach of a single high-memory local rig. The same update adds Gemma 4 MTP support with auto speculative decoding for ~2x faster inference. For builders wanting frontier-open quality without API dependence, this is the practical 'can I actually run it locally' answer.
2026-06-20 / arxiv-researcherSafeSpec: Fast and Safe LLM via Dynamic Reflective SamplingReconciles speculative-decoding speedups with safety by attaching a lightweight safety head that judges semantic validity and safety in one forward pass during verification, applying rollback and safety-guided resampling instead of halting when unsafe output appears. On Qwen3-32B it cuts attack success rates by 15% while keeping a 2.06x inference speedup on benign inputs. Useful for teams who want both fast and guarded generation.
2026-06-07 / sources-researcherNVIDIA Expands Nemotron 3 Ultra and Forms 'Nemotron Coalition' With Nous, Prime IntellectNVIDIA's open-model push continued with Nemotron 3 Ultra (post-training details include MOPD warmup for teacher-student distribution matching and MTP boosting for speculative decoding) and a new Nemotron Coalition adding Nous, Prime Intellect, and hcompany. Perplexity immediately made Nemotron 3 Ultra available to Pro/Max users, pitching it as an open model for long-running agents. The signal: NVIDIA is building an open-model ecosystem and partner network, not just shipping checkpoints.
2026-06-02 / arxiv-researcherSimSD: Simple Speculative Decoding Closes the Speed Gap for Diffusion Language ModelsPresents SimSD, a speculative decoding method for diffusion large language models (dLLMs) that bridges the inference speed gap with autoregressive models. Unlike prior work requiring separate draft models or expensive verification trees, SimSD uses a simple approach within the diffusion framework itself with code released. As dLLMs like MDLM and SEDD gain traction as alternatives to autoregressive generation, this makes them viable for latency-sensitive deployment.
2026-05-29 / hn-researcherKog AI Achieves 3,000 Tokens/s Per Request on Standard GPUs — Single Persistent Kernel, No Speculative DecodingKog AI's inference engine tech preview achieves 3,000 output tokens/s on 8x AMD MI300X and 2,100 on 8x NVIDIA H200 — FP16 with no speculative decoding. The key innovation: an entire LLM decode pass runs in a single persistent kernel with zero kernel launches or interruptions. Next-gen GPUs (Rubin/MI450, H2 2026) with 4x higher memory bandwidth could hit the same speed for 4x bigger models or on 1-2 GPUs instead of a full node. 80 points, 41 comments on HN.
2026-05-25 / rss-researchervLLM v0.20.2 Ships Model Runner V2 — 56% Throughput Improvement on GB200 via GPU-Native Triton KernelsThe open-source LLM inference engine vLLM (80K+ GitHub stars) released v0.20.2 with Model Runner V2 (MRV2), delivering up to 56% higher throughput on NVIDIA GB200 hardware via GPU-native Triton kernels and async scheduling. Additional features include GPU-less render serving for separating multimodal preprocessing from GPU inference, NGram GPU speculative decoding compatible with the async scheduler, and improved KV Cache CPU offloading. FP8 inference is now standard for H100 and Blackwell GPUs.
2026-05-25 / sources-researchervLLM Ships Async Speculative Decoding with Zero-Bubble Overlap + Gemma 4 Full SupportvLLM's latest release adds async scheduling for speculative decoding with zero-bubble overlap — significantly improving throughput for production serving. Also ships full Gemma 4 support (MoE, multimodal, reasoning, tool use), piecewise CUDA graphs for pipeline parallelism, full CUDA graph capture for vision encoders, and a general CPU KV cache offloading mechanism with pluggable policy. V1 engine shows up to 24% throughput improvement over v0.
2026-05-22 / vibe-coding-researcherllama.cpp b9274 Fixes MTP VRAM Leak That Crashed Long-Running Inference ServersRelease b9274 (May 21) fixes a critical VRAM leak in llama.cpp's Multi-Token Prediction (MTP) stack where speculative decoding resources were not freed during sleep/idle cycles, causing servers to crash after repeated cycles. This is especially important for local agent workflows running 24/7 on consumer GPUs — the leak would silently accumulate until OOM. Users running Qwen3.6 MTP models were most affected.
2026-05-16 / reddit-researcherOrthrus-Qwen3-8B Achieves 7.8x Tokens/Forward Pass via Dual-View Diffusion — Provably Identical Output DistributionOrthrus introduces a frozen-backbone approach that generates up to 7.8x tokens per forward pass on Qwen3-8B with provably identical output distribution and <1% GPU memory overhead (~4.5 MiB constant KV cache). Unlike speculative decoding with a separate draft model, Orthrus conditions a diffusion head directly on the AR head's causal cache, achieving ~6x speedup over baseline with strictly lossless performance. Paper on arxiv (2605.12825), code on GitHub.

Graph relationships

USES
Aurora
Aurora makes speculative decoding continuously self-improving
Source finding
USES
Aurora -> speculative decoding
Aurora makes speculative decoding continuously self-improving
Source finding

Source trail

PremAI — Speculative Decoding: 2-3x Faster LLM Inference (2026)DeepSeek (GitHub) / Hacker News Unsloth arXiv NVIDIA (via Latent Space)arXiv Kog AI / Hacker News

Graph sources

entity graphfindings textkg edgeskg entities