Fetching from the wire…
Infra2026-09-03 · source-backed
Build b10781 fixes PR #28190: the FA dequant path skipped engagement because of an nb[3] stride check the shader never reads when ne[3] == 1. Cache views carry the full-buffer stride there, so the check collapsed to n_kv == kv_size and the optimized path only turned on with the cache completely full. Anyone who benchmarked Vulkan flash attention on llama.cpp before this build was measuring the unoptimized path for nearly every real workload. Build b10782 separately unblocks CUDA graph optimization for multi-GPU under GGML_CUDA_GRAPH_OPT=1.
Each link below shares sources, entities, or timing with this story.
Build b10677 fixes ggml_vk_graph_optimize, where is_src_of didn't treat two views of one tensor as dependent, so the optimizer reordered nodes across aliased reads and writes. Maintainers describe the result as silently wrong tokens under greedy decoding, different output on e...
Build b10758 (September 2) fuses matmuls landing on Hexagon HMX, fuses MUL_MAT_ID into MUL_MAT_ID_NX, and adds VA defragmentation so large-dim runs abort less on fragmented address space (release). Build b10757 handles batch sizes above 4 for IQ3_S mat-vec when NUM_COLS > 4, r...
Build b10715 folds the encoder into the decoder's embd branch so one llama_decode handles it. The encoder is a fully-connected layer plus a norm, but running it as a separate llama_encode forced a round trip before the injection decode could re-upload it, and built a second gr...
Build b10687 defaults the xmem F16xF32 GEMM on for Adreno X2E after measuring kernel_mul_mm_f16_f32_l4_lm running gpt-oss-20b attention projections at about a quarter the speed of the tuned dense q4_0 GEMM. Worth 25% prefill. The route already existed but was opt-in, so nobody...
PR #19378 landed in llama.cpp this week, and I think most people are underselling what it means. Backend-agnostic tensor parallelism via --split-mode tensor makes multi-GPU inference work across AMD, Intel, and Apple Silicon. Not just CUDA. Everything. For context: llama.cpp h...
Cross-platform desktop STT built with Tauri. Fully offline using Whisper and Parakeet models. GPU-accelerated on CUDA, or CPU-only via Parakeet V3. Designed to be *"the most forkable speech-to-text app."* Competes with paid tools like Wispr Flow with zero cost and full privacy...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.