Motif 3 Technical Report: a 314B-Total / 13.2B-Active MoE With 384 Routed Experts, Trained on 12.5T Tokens With a New Grouped Differential Latent Attention

arXiv / HuggingFace Daily Papers·medium signalXBlueskyLinkedInCopy link

arXiv 2608.09119 (Aug 10), from 27 authors led by Junghwan Lim and Joon Son Chung, documents a sparse MoE with 314B total parameters activating 13.2B per token, selecting 8 of 384 routed experts per layer, pretrained on ~12.5T tokens of web/STEM/code/math/multilingual data at up to 256K context. The architectural contributions worth noting are Grouped Differential Latent Attention (GDLA), which fuses grouped differential attention with Multi-head Latent Attention's compressed KV representation, plus manifold-constrained hyper-connections, Expert-Specific PolyNorm activations, and multi-token prediction; training used MXFP8 compute and window-aware context parallelism. Post-training is unusual: six specialist RL teachers plus a supervised software-engineering teacher, combined via multi-teacher on-policy distillation.