Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-16 · high
A small on-device model drafts every token, and the cloud model only steps in to fix a wrong guess, no retraining needed.
Why now: This surfaced in the 2026-08-16 briefing, which calls the 76% cut the most direct cost lever for local-first products paying per cloud token.
SPADE cuts cloud model calls 76%, per a new arXiv paper on speculative decoding. For products that pay per cloud token, this is the most direct cost lever around.
Speculative decoding normally runs a draft model and a verifier model on one machine. SPADE splits them across devices instead. The draft model moves to the edge, and the cloud keeps the large verifier. The edge model proposes tokens locally. The cloud model checks them in parallel batches and only corrects a token when the draft model guessed wrong. Neither model needs retraining, and the method is plug-and-play, per the paper.
On SpecBench and CNN/DailyMail summarization tasks, the split cut cloud calls 76% with no drop in accuracy, the paper reports. The cloud verifier still checks every proposed token, so output quality doesn't change even as call volume drops.
Both benchmarks are structured tasks. The paper doesn't say whether the reduction holds on messier, more varied production traffic, the kind most real products actually see.
Each link below shares sources, entities, or timing with this story.
Same source domain / Shared topic / Tension / Downstream implication
Reported by the same outlet (arxiv.org); overlapping topics (against, correction); pushes against this story (against).
CNN criticizes Perplexity / Shared topic
Linked by a graph relationship (CNN criticizes Perplexity); overlapping topics (call, cloud, model).
Shared entity: Speculative / Shared topic / Earlier coverage
Both cover Speculative; overlapping topics (cloud, model, token); earlier Speculative coverage from 2026-05-22.
Same source domain / Shared topic / Tension
Reported by the same outlet (arxiv.org); overlapping topics (accuracy, against, cost); pushes against this story (against).
Reported by the same outlet (arxiv.org); overlapping topics (boundary, cuts, token); pushes against this story (versus).
Reported by the same outlet (arxiv.org); overlapping topics (accuracy, against, call); pushes against this story (against).
Reported by the same outlet (arxiv.org); overlapping topics (against, call, cost); pushes against this story (against).
Reported by the same outlet (arxiv.org); overlapping topics (against, call, model); pushes against this story (against).