Public story · 2026-08-31 · high

llama.cpp folds speculative encoder into decoder pass

The fix cuts a device-to-host round trip per decode round, and the commit lists an AI model as co-author alongside a human contributor.

Why now: Covered in llama.cpp's August 31 release notes.

Issue

2026-08-31

Confidence

high

Sources

Redaction

passed

Story

llama.cpp folded its DFlash speculative encoder into the decoder's embedding branch, so one llama_decode call now handles both steps. Speculative decoding exists to skip redundant passes through the model, and forcing a device round trip on every round worked against that.

Before this change, the encoder ran as its own llama_encode call. It's a small op, just a fully-connected layer plus a norm. But running it separately forced a round trip from device back to host before the injection decode could re-upload the result. It also meant building a second computation graph every round.

Merging the two removes that round trip and the extra graph.

Claude Fable is listed as co-author on the commit, alongside a human contributor, per llama.cpp's b10715 release notes. This is a patch to how tensors move between decode calls, credited jointly to a human and an AI model. The notes don't say how the work split between the two, or what review it went through before merging.

For anyone running speculative decoding setups on llama.cpp, the result is fewer device syncs per round. That should show up as lower latency on anything using DFlash.

Source trail

Build b10715 More from github.com

Entities

Build Claude Fable DFlash

Claim evidence

llama.cpp folds speculative encoder into decoder pass
Build b10715

Provenance

Canonical issue: 2026-08-31
AI generated: yes
Story unit: 2026-08-31-llama-cpp-fused-the-dflash-speculative-encoder-into-kv-cache-injection-cutting-a-device-to-host
Labels: source-backed, canonical briefing excerpt

Fetching from the wire…

← The Wire

Public story · 2026-08-31 · high

llama.cpp folds speculative encoder into decoder pass

The fix cuts a device-to-host round trip per decode round, and the commit lists an AI model as co-author alongside a human contributor.

Why now: Covered in llama.cpp's August 31 release notes.

Issue

2026-08-31

Confidence

high

Sources

Redaction

passed

Story

Merging the two removes that round trip and the extra graph.

For anyone running speculative decoding setups on llama.cpp, the result is fewer device syncs per round. That should show up as lower latency on anything using DFlash.

Source trail

Build b10715 More from github.com

Entities

Build Claude Fable DFlash

Claim evidence

llama.cpp folds speculative encoder into decoder pass
Build b10715

Provenance

Canonical issue: 2026-08-31
AI generated: yes
Story unit: 2026-08-31-llama-cpp-fused-the-dflash-speculative-encoder-into-kv-cache-injection-cutting-a-device-to-host
Labels: source-backed, canonical briefing excerpt

llama.cpp folds speculative encoder into decoder pass

Story

Related stories

Anthropic Engineers Say Stop Building Agents, Build Skills Instead

Meta ships Muse Glimmer, a 30B open agent model

Jensen Huang's First Tweet Doubles AI Coalition to 50

Kimi K3 Beats Fable 5 on Frontend Code, Ships July 27

A model wrote the fastest megakernel anyone's submitted, and beat every other frontier model doing it

Build structured agent loops with small local models (Qwen 3.6 32B, Gemma 4 9B).

Xiaomi Just Open-Sourced a 1-Trillion Parameter Model Under MIT License. It Built a Compiler in 4 Hours.

Kimi K2.6: A 1-Trillion-Parameter Open-Weights Model Just Beat GPT-5.4 on SWE-Bench Pro

Source trail

Entities

Claim evidence

Provenance

llama.cpp folds speculative encoder into decoder pass

Story

Related stories

Anthropic Engineers Say Stop Building Agents, Build Skills Instead

Meta ships Muse Glimmer, a 30B open agent model

Jensen Huang's First Tweet Doubles AI Coalition to 50

Kimi K3 Beats Fable 5 on Frontend Code, Ships July 27

A model wrote the fastest megakernel anyone's submitted, and beat every other frontier model doing it

Build structured agent loops with small local models (Qwen 3.6 32B, Gemma 4 9B).

Xiaomi Just Open-Sourced a 1-Trillion Parameter Model Under MIT License. It Built a Compiler in 4 Hours.

Kimi K2.6: A 1-Trillion-Parameter Open-Weights Model Just Beat GPT-5.4 on SWE-Bench Pro

Source trail

Entities

Claim evidence

Provenance