Fetching from the wire…
01
02
03
04
05
06
07
08
Infra2026-09-15 · source-backed
PR #28789, merged September 15, found that ggml_backend_rpc_buffer_set_tensor hashed every transfer above the 10 MB threshold and let rpc-server -c serve it from a file cache intended for weights, which also caught the activations ggml_backend_sched copies between backends (GitHub). With a two-node split of Qwen3.8-Flash-Next, every prefill ubatch over 10 MB got hashed and written; after a day the cache held 14,802 files and 1.4 TB. Once the disk filled, the server wrote truncated cache files and later served them as complete, producing Inf/NaN activations, which is how anyone noticed. Fix gates hashing on GGML_BACKEND_BUFFER_USAGE_WEIGHTS, adds a cache_flag byte to SET_TENSOR, bumps RPC_PROTO_MAJOR_VERSION to 7. A 16 MiB compute tensor drops from 19 ms to 2.9 ms.
Each link below shares sources, entities, or timing with this story.
Released September 4, it adds llama_lazy_mode / --lazy-mode for on-demand tensor reading (#27794), a max_buf_size quantize parameter capping quantizer RAM (#27795), quantizer row-slab streaming (#27830), and a fix preventing RAM peaking during load (#27483). It also adds spars...
MIT, Swift and MLX, created August 28, 220 points on Hacker News (GitHub). It runs Qwen3.8-Flash-Next, 103.8GB across 24 files at 4-bit, by loading only the 3.8GB dense trunk into RAM (about 2 second startup) and reading routed experts with pread into a fixed pool of cache slo...
The repo appeared August 24, opening the weights of a multimodal MoE that Qwen frames explicitly as an architecture preview, the same role Qwen3-Next played for Qwen3.5 (GitHub). The hybrid Gated DeltaNet plus Gated Attention design it previews already carried through the Qwen...
Released August 27 with GGUFs for both, claiming 5x faster inference for RAM offloading, working repeated compaction, chats that recover after disconnects instead of losing the reply, and memory estimates shown before a load (GitHub). That's roughly a 24-hour turnaround from t...
PR 27742 adds gated delta net layers, 512-expert MoE with top-10 selection, hyper-connections, query-key sparse attention and the per-layer n-gram embeddings as a mmap table that can sit in RAM or on disk (GitHub). Reported 55 tok/s on 4x3090 with the Q4 GGUF, and one commente...
Every flaky quantized agent I've debugged, I blamed the quantization. A paper published September 4 says I've probably been blaming the wrong layer (arXiv 2609.04748). The setup is clean. Model, decoding parameters, seed, request order and batch size all fixed. Requests issued...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.