Fetching from the wire…
01
02
03
04
05
06
07
08
Infra2026-09-19 · source-backed
PR #50045 stops slow secondary KV tiers (disk, shared storage, P2P) from accumulating unbounded job queues. An exponential moving average tracks per-tier store completion latency, a tier enters pressure above a 1.0s high-water mark and clears below 0.5s, and stores to a pressured tier drop while loads keep working. Two Prometheus metrics per tier make the drop visible instead of silent. vLLM
Each link below shares sources, entities, or timing with this story.
Speculative decoding (a small draft model proposes tokens the target verifies in parallel) gives 2-5x latency wins, but only in memory-bound, low-batch regimes. At large batch sizes the GPU is already compute-bound, and the extra draft-and-verify work makes inference *slower*...
oMLX runs local LLMs on Apple Silicon with a two-tier KV cache: hot cache in RAM, cold cache on SSD in safetensors format. When a previous context prefix recurs, blocks restore from disk instead of recomputing. Time-to-first-token drops from 30-90s to 1-3s on long contexts. 13...
His September 12 post calls Astra a bigger jump than Fable 5 to 5.1, citing 98% on FrontierMath Tier 4, 98.1% on extended NYT Connections against Fable 5.1's 90%, the first autonomous Montezuma's Revenge clear and a one-shot Portal completion. The monitorability problem is Nee...
The single feature is normalizing mid-stream read errors (#3827), so a failure surfaces as a consistent SDK exception rather than whatever the transport threw (GitHub). The fixes clear a long backlog: vector store file polling is bounded (#3401), API error codes normalized to...
PR #28789, merged September 15, found that ggml_backend_rpc_buffer_set_tensor hashed every transfer above the 10 MB threshold and let rpc-server -c serve it from a file cache intended for weights, which also caught the activations ggml_backend_sched copies between backends (Gi...
Agent Manager now claims a ready worktree instead of running a full checkout, on by default at the cost of one extra checkout of disk per open project (GitHub). The primary checkout resolves in one git call instead of four, agents and skills are discovered for a new worktree b...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.