Fetching from the wire…
01
02
03
04
05
06
07
08
Vibe Coding2026-09-10 · source-backed
Part 4 of a running 2x3090 series: prefill was 80+ seconds to first token on an 8k prompt and 24 minutes on a 119k one, and releasing the 150-slot expert cache off the GPU during prompt processing bought the speedup. The comments are why it's here. A reader calculated the 2.8s release/restore only pays back past about 500 prefill tokens, the author checked his own long-document run and found 4.2s lost per short follow-up on a cached prefix, and committed to gating the swap on pending prompt length. Public self-correction with numbers, which is rarer than the optimization.
Each link below shares sources, entities, or timing with this story.
Supports both an integrated MTP head and a separate -md draft file, with the poster reporting 45 to 90 tok/s on a 5090 with 128GB system RAM and working configs down to a 12GB 4070. A separate 2x3090 plus DDR4 build reports 37-41 t/s decode with UD-Q4_K_XL plus expert cache pl...
Every flaky quantized agent I've debugged, I blamed the quantization. A paper published September 4 says I've probably been blaming the wrong layer (arXiv 2609.04748). The setup is clean. Model, decoding parameters, seed, request order and batch size all fixed. Requests issued...
A practitioner running 2x Strix Halo 128GB over USB-C 4 with llama.cpp RPC compared both models at Q8_K_XL on real coding work (r/LocalLLaMA). A task Qwen finished in 25 minutes on medium took DeepSeek 12. The poster attributes the gap to fewer hallucinated detours. The sharpe...
A month of solo work produced quants for LongCat-Flash-Lite-Sparse, Qwen3.8-27B, Qwen3.5-122B-A10B with MTPs, Qwen3-Coder-Next and Laguna-S2.1 with vision. Getting there meant writing Heretic support for the architecture from scratch and then adding llama.cpp support, and main...
The video post shows the 80GB model generating on a 12GB mid-range handset with aggressive quantization on the dense part plus unnamed optimizations. The CPU hits 80C, which the top commenter flags immediately, so this demonstrates what fits rather than something you'd leave r...
Unsloth's UD-Q2_K_XL (78.9 GB) plus a 358,400-token slot via YaRN from the native 262,144 with fp16 KV fits under the default 96GB Metal wired limit, no sysctl hack. Cold prefill runs 1,561 t/s at 5.6K context down to 318 t/s at 111K; a normal incremental turn is 77-854 t/s ou...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.