halogen 0.12.0 pushes Qwen3.8-Flash-Next to 38.3 tok/s decode at one million tokens of context on a Strix Halo box

GitHub / peonist-ai (via r/LocalLLaMA, 97 upvotes)·medium signalXBlueskyLinkedInCopy link

The halogen maintainer fixed context-depth degradation and published a same-machine, same-session 0.11.10 vs 0.12.0 comparison on a Ryzen AI Max+ 395 with 128 GB: decode at 1,004,581 tokens went 27.3 to 38.3 tok/s, prefill 790 to 937 tok/s (21.2 down to 17.9 minutes cold), and at 258,794 tokens decode moved 42.9 to 45.0. The 32k served mean was unchanged, so the gain is specifically at depth. Running at 1M needs `HALOGEN_ROPE_YARN=4 HALOGEN_CTX=1048576` and the 128 GB machine; a follow-up turn over the prompt cache reaches first token in about 0.55s.