Fetching from the wire…
Infra2026-09-21 · source-backed
PR #45635 adds routed-expert output as the first auxiliary type, and the design note explains why the obvious approach fails: R3 can't be a request-local GPU buffer once prefix caching and async scheduling are on, because a cache hit skips execution, GPU slots aren't stable identities, and speculative decoding executes rows that get rejected. Storing immutable R3 blocks under KV-compatible hashes lets KV and R3 for the same prefix be reused together. Ships a bounded local shared-memory backend, with distributed storage named as follow-up.
Each link below shares sources, entities, or timing with this story.
A llama.cpp fork by fewtarius aimed at AMD APUs, iGPUs and handhelds adds a persistent SSD-backed KV cache with hot/warm/cold tiering. On an Ayaneo Flip KB running Qwen3.6-35B over a 15,700-token prompt, cold TTFT was 143.1 seconds versus 0.99 seconds warm, a 144.5x speedup. A...
JustVugg/colibri, Apache-2.0, 27,290 stars, treats storage, RAM and VRAM as one inference hierarchy with zero engine dependencies and one C file per model family. Eight families work today, including Kimi K3 at 2.8T and Inkling at 975B, all behind the same coli chat / coli ser...
The abliteration tool gained 215 stars to reach 30,103, but the stronger signal is downstream: the HF trending endpoint returns DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NM-DAU and Momoking/Qwen3-VL-32B-Heretic-MiniMax-H3-NVFP4, both naming the too...
PR #26622 pushes a user-specified number of FFN sublayers to CPU while attention stays on the GPU, mirroring --n-cpu-moe (GitHub). The practical note from the r/LocalLLaMA thread is that the previous route was regex matching in -ot style, and for a model you plan to run for mo...
arXiv 2607.29678, from Zhenyu Zhang and Zhichao Cao, profiled 153,951 real calls across two agent ecosystems and found that *even with prompt KV caching*, frontend re-tokenization of the full transcript dominates TTFT at high cache hit rates. Nobody instruments this. TokTier d...
Alagappan Valliappan shows built-in Multi-Token-Prediction draft heads run full attention over the entire KV cache at every draft step, so at million-token context the "negligibly cheap" draft dominates cost and deep native drafts can go net-negative. Applying a StreamingLLM-s...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.