Entity trail

speculative decoding

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Showing the first 40 findings. More graph evidence exists in the corpus.

Corpus findings

2026-08-25 / github-pulse-researcherspec-ptc applies speculative decoding thinking to tool calls instead of tokensalexzhang13/spec-ptc, created 2026-08-21 and at 84 stars, implements speculative programmatic tool calling for harnesses including RLM and CodeAct, and was still being pushed on 2026-08-24. The idea transplants the draft-and-verify pattern from token generation up to the tool-call layer, where the latency cost per wrong guess is far higher than a discarded token. Single-source and early, but it names a specific optimization target that agent harness authors have mostly left alone.
2026-08-25 / reddit-researcherJetBrains Chose Qwen 3.6 Over 3.8 Because 3.8 Needs Reasoning, and Then Cut Generation Time in Half With MTP Plus N-Gram Speculative DecodingThe companion engineering post explains that Qwen 3.8 degrades badly with reasoning off, and enabling it generates about 5x more tokens for a roughly 4x slowdown, which killed it for local M5 use. JetBrains disabled reasoning entirely on 3.6 for 2-3x fewer tokens and about 2x speed, used 4-bit over 8-bit for another 2x, applied 8-bit arithmetic during prefill on M5 for about 40% faster prefill, and combined Multi-Token Prediction with n-gram speculative decoding for a further 2x on generation. They also extended rolling context to reuse KV-cache across tasks so the agent stops re-reading files, and disabled multi-agent mode outright.
2026-08-24 / arxiv-researcherTreeWY Removes Per-Node State Snapshots From Speculative Decoding on Gated DeltaNet HybridsTreeWY (arXiv 2608.20961, Aug 21) attacks a specific cost in serving hybrid models where most layers are linear-attention Gated DeltaNet carrying a fixed-size recurrent state: to verify and roll back draft tokens, current systems snapshot the full recurrent state at every draft position, and those snapshots cannot be shared across draft-tree branches. Using a tree-structured WY transform of the gated delta rule, it computes every draft node's output with a single triangular solve and reconstructs only the accepted state on commit, storing a small pseudo-value matrix instead. On Qwen3.5 35B and 397B serving benchmarks this cuts speculative state memory and KV-cache pressure at identical acceptance length, raising throughput and lowering time-to-first-token where memory binds, at a few percent cost where it does not.
2026-08-24 / reddit-researcherA Full NVFP4 vLLM Recipe Fits Qwen3.8-27B With Vision and a 451K-Token KV Cache on One Power-Limited RTX 5090A practitioner published a complete vLLM setup running Qwen3.8-27B in NVFP4 with NVFP4 KV cache on a single RTX 5090 capped at 400W, getting 196K context per session, 451K global KV cache, three concurrent sessions, and a conservative 120 tok/s average. Their llama-benchy 0.4.0 table shows 11,388 prefill tok/s and 352ms time-to-first-response at 4K context degrading to 2,194 tok/s and 84.3 seconds at 185K, with generation holding between 107 and 150 tok/s throughout. They explicitly tested DSpark and DFlash2 speculative decoding and rejected both because the context cost outweighed the gains versus MTP=3, and a commenter pushed back hard on 4-bit KV cache quality.
2026-08-23 / github-pulse-researcherMTPLX gets 3x MLX speedups on Qwen 3.8 27B using the model's native multi-token prediction head, no draft model neededyoussofal/MTPLX implements native MTP speculative decoding on Apple Silicon, reporting roughly 3x faster generation on Qwen 3.8 27B without any external drafter model. Eliminating the separate draft model removes the usual memory overhead and the draft-target alignment tuning that makes speculative decoding awkward on memory-constrained Macs. At 1,585 stars since 2026-05-02 and pushed 2026-08-22, it is smaller than the mainline MLX projects but targets a specific bottleneck they have not closed.
2026-08-22 / github-pulse-researcherA single DGX Spark now serves DeepSeek V4 Flash at 3.0bpw with a 439,622-token KV pool, no second nodeMiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark (created 2026-08-20, MIT, 116 stars and 15 forks) is a Docker recipe serving the 3.0 bpw EXL3 build of DeepSeek V4 Flash 0731 on one GB10 DGX Spark at tensor-parallel 1, where the official FP4 build needs TP2 across two Sparks. Defaults changed on 2026-08-21 to a deep-context NVFP4 config: native 432-byte KV records, 0.94 GPU memory utilization, 384,000 max model length, DSpark K5 speculative decoding with a K64 draft, yielding a 439,622-token KV pool and a 370,104-token context stress-tested with exact needle recall. Weights are ~107 GB and download locally by default.
2026-08-21 / rss-researcherLiquid AI's DSpark draft models hit 3.18x throughput on GPU and cut function-calling latency 57%LFM2.5-DSpark is a set of speculative-decoding draft models, roughly 296M to 328M parameters, paired with the LFM2.5 1.2B, 2.6B and 8B-A1B targets. Liquid reports up to 3.18x throughput on GPU and 2.87x on-device, with the 2.6B hitting 2.67x on an H100 and 2.27x on an M4 Max MacBook, and function-calling latency down 57% on average. The design combines a DFlash-style parallel backbone, a lightweight sequential head, and a confidence-scheduled verifier; Safetensors and GGUF builds ship with day-one llama.cpp and SGLang support.
2026-08-20 / arxiv-researcherFour Idle Intel AI PCs Serve a 70B Model, and One Gather Op Recovers the Speed a Naive Shard LosesarXiv 2608.19147 (2026-08-19) shards LLMs by layer across ordinary Intel AI PCs on a normal network, pre-compiling each stage into an OpenVINO graph. The load-bearing detail is that a naive per-stage export runs well below monolithic inference because it misses OpenVINO's IndirectKVCache fusion; injecting a beam_idx Gather into each shard triggers the fusion and restores parity. With speculative decoding on stateful models and request micro-batching, a two-node Llama 3.1 8B INT4 pipeline serves two concurrent users at 1.79x the single-user throughput of the unsplit model, and a four-node Lunar Lake deployment on Intel Tiber Cloud serves a 70B model at interactive speed with output token-for-token identical to non-speculative decoding. Code and reproduction scripts are at github.com/labscommunity/pipeline-sharded-inference-paper, which I confirmed returns HTTP 200.
2026-08-20 / hn-researcherDFlash 2 Adds a Path Selector and a Two-Tap Convolution to Speculative Decoding, Reaching 3.4x Throughput on Qwen3.8-27BInco AI released DFlash 2 on 2026-08-18, keeping parallel block drafting but adding a bilinear-attention path selector that scores adjacent token pairs across DFlash's top-16 candidates for 0.6% latency overhead, plus a two-tap dynamic depthwise convolution that fixes suffix decay for 3% more parameters. The result is a 21% gain in mean acceptance length over DFlash, 2.7-3.4x throughput versus autoregressive decoding on Qwen3.8-27B, and 3.1-4.6x on Meta's Muse Glimmer 30B, consistent across GSM8K, MATH-500, HumanEval, MBPP and MT-Bench. Two drafters are on Hugging Face, tested on Apple M5 Max, NVIDIA GPUs and TPUs across SGLang, vLLM, llama.cpp, Ollama and oMLX.
2026-08-18 / hn-researcherQwen3.8 27B Hits 50.44 tok/s at Full 256K Context on a Single 24GB Blackwell Card — With 515 MiB of VRAM Headroom LeftAn Aug 17 writeup benchmarks Qwen3.8 27B on an RTX PRO 4000 Blackwell SFF using a custom 5.01 BPW iMatrix/NVFP4 hybrid quant (bulk matrices NVFP4, sensitive tensors Q5_K/Q6_K) and a patched llama.cpp with Gated DeltaNet support and chained MTP speculative decoding. Production mean was 50.44 tok/s versus 21.19 tok/s target-only greedy — a 2.81x speculative speedup — validated at 261,500 input tokens, occupying 23,952 of 24,467 MiB. The honest caveats matter: throughput collapses to 12.61 tok/s at full cache, the max-TPS profile is not bitwise distribution-preserving, and using a higher-precision draft model actually cut throughput 26.6%.
2026-08-17 / reddit-researcherKoboldCpp v1.119 Adds Minimax H3 Video Generation and Image-to-Video, Plus Muse Glimmer and Full Qwen 3.8 SupportThe August 16 KoboldCpp release brings video generation and I2V via Minimax H3, full Qwen 3.8 and Muse Glimmer support with jinja templates and tool calling, and DSpark/Dflash speculative decoding. Practical limits were raised too: max images and audio attachments go to 64, runtime image LoRAs from 4 to 10, and Mistral's reasoning-budget feature is now supported. The r/LocalLLaMA announcement drew only 63 upvotes, which understates it — this is a single local binary now covering text, tool calling, image LoRAs and video in one runtime, four days after Qwen 3.8's release.
2026-08-16 / arxiv-researcherSPADE Cuts Cloud Model Calls 76% With Zero Accuracy Loss by Running the Draft Model on the EdgeSPADE (arXiv 2608.13076, 2026-08-13) splits speculative decoding across the edge/cloud boundary rather than within one machine: a compact draft model on the edge proposes tokens, a large cloud verifier validates them in parallel, and only rejections trigger a cloud correction. The design is plug-and-play with no retraining and shifts the bulk of compute to the edge. On SpecBench and CNN/DailyMail across multiple NLP tasks it reduces cloud model calls by 76% with zero accuracy loss versus the full model — a direct cost lever for anyone paying per cloud token in a local-first product.

Graph relationships

USES
Aurora
Aurora makes speculative decoding continuously self-improving
Source finding
USES
Aurora -> speculative decoding
Aurora makes speculative decoding continuously self-improving
Source finding

Source trail

GitHub JetBrains Junie blog arXiv 2608.20961 r/LocalLLaMA (46 upvotes), full write-up at gist.github.com/co-l/c2aeaf40b53fcacfe9dd3293be75f23a GitHub Trending GitHub Hugging Face Blog (Liquid AI)arXiv 2608.19147 Inco AI / Hacker News piszczek.pl / Hacker News r/LocalLLaMA / GitHub LostRuins/koboldcpp arXiv 2608.13076

Graph sources

entity graphfindings textkg edgeskg entities

Fetching from the wire…