Qwen3.8-Flash-Next Matches a 397B Predecessor on 8 of 14 Benchmarks Using a Ninth of the Training FLOPs

arXiv 2608.30320·high signalXBlueskyLinkedInCopy link

The architecture report describes a 125B sparse mixture-of-experts model activating 6B parameters per token, plus 51B of n-gram embedding tables held off the accelerator in host memory with prefetching. Against the prior 397B-A17B model it leads on 8 of 14 pre-training benchmarks and trails by at most 2.6 points elsewhere, using a third of the activated parameters, a third of the training tokens and roughly a ninth of the training FLOPs. The design mixes Gated DeltaNet with one full-attention layer per four, adds Qwen Sparse Attention for continued pretraining and a four-branch Gated Residual stream, and reports that pairing it with Muon shifts optimal learning rate and batch size upward and removes the need for batch-size warmup.