Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-09-20 · high
Decode speed at a million tokens of context rose from 27.3 to 38.3 tokens a second on the same 128 GB machine.
Why now: This comparison is dated September 2026 and covers only the one 128 GB configuration tested.
Decode speed moved from 27.3 to 38.3 tokens a second at 1,004,581 tokens of context in halogen 0.12.0, per the halogen-flash-server repo.
For anyone running local models past a few hundred thousand tokens of context, that's a real wait-time difference. Cold prefill time on the same rig fell from 21.2 minutes to 17.9.
The comparison ran on the same Ryzen AI Max+ 395 machine and the same session. Prefill throughput at that depth moved from 790 to 937 tokens a second. Reaching those numbers needs HALOGEN_ROPE_YARN=4 and HALOGEN_CTX=1048576 set, plus enough memory to hold a million tokens of context in the first place.
Yes, but the fix is specific to depth. At 258,794 tokens, decode only moved from 42.9 to 45.0 tokens a second. The speed served at 32,000 tokens didn't change at all.
A separate change lets the server reuse the prompt cache across turns, bringing first-token latency to about 0.55 seconds.
The numbers come from one machine. Whether the fix helps on a different GPU or a different context-length curve isn't reported.
Each link below shares sources, entities, or timing with this story.
June local-inference benchmarks across the 128GB class put NVIDIA's DGX Spark (~$4k), AMD's Strix Halo / Ryzen AI Max+ 395 (~$2 to 3k), and the M5 Max 128GB (~$5k) head to head (Hardware Corner). Prompt processing favors CUDA hard. But token generation lands at a surprisingly...
A llama.cpp fork by fewtarius aimed at AMD APUs, iGPUs and handhelds adds a persistent SSD-backed KV cache with hot/warm/cold tiering. On an Ayaneo Flip KB running Qwen3.6-35B over a 15,700-token prompt, cold TTFT was 143.1 seconds versus 0.99 seconds warm, a 144.5x speedup. A...
The 0731 checkpoint sits at 1,048,685 downloads and 3,112 likes, second on HuggingFace trending behind MiniMax-H3. For scale, Kimi-K3 has been live since June 13 and sits at 1,565,484, so Flash covered two-thirds of that in under two weeks. Small fast-inference tiers are where...
10,936 tokens against 1,552,491 for raw HTML across 15 real pages, and the author claims it was the only reader returning real content on every page tested. The concrete cost comparison: $0.27 against $0.52 for five Wikipedia questions, 23% cheaper than WebFetch and 35% cheape...
rust-v0.150.1, published August 27, fixes remote compaction ignoring retained images when computing token budget; it now trims older images as needed (GitHub). Sessions passing screenshots to a vision-capable agent were budgeting as though those images cost nothing, which prod...
MIT, Swift and MLX, created August 28, 220 points on Hacker News (GitHub). It runs Qwen3.8-Flash-Next, 103.8GB across 24 files at 4-bit, by loading only the 3.8GB dense trunk into RAM (about 2 second startup) and reading routed experts with pread into a fixed pool of cache slo...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.