Fetching from the wire…
01
02
03
04
05
06
07
08
Research2026-09-09 · source-backed
Across six activated-parameter scales, models up to 6B total non-embedding parameters, about 20 trillion tokens and 200,000 equivalent H800 GPU-hours: optimal batch size follows a power law in training tokens, optimal learning rate scales with compute, and activation ratio enters both as an additional multiplicative power-law factor. On a held-out 12B-total MoE activating 1/64 of its experts, predicted hyperparameters land close to observed optima. arXiv 2609.08690
Each link below shares sources, entities, or timing with this story.
The architecture report describes a 125B sparse MoE activating 6B parameters per token, plus 51B of n-gram embedding tables held off the accelerator in host memory with prefetching (arXiv 2608.30320). Against the prior 397B-A17B model it leads on 8 of 14 pre-training benchmark...
Measuring expert time on two datacenter GPU generations shows it is linear in neither token count (EPLB, LPLB, UltraEP) nor activated-expert count (METRO). Below roughly 156 to 168 tokens, HBM weight streaming dominates so cost attaches to activated replicas. Above it, grouped...
Most looped-transformer results compare at fixed model size, conflating architecture with extra compute. SMELT matches per-token FLOPs, non-embedding parameters and KV cache against an unlooped baseline, scaling to 54B non-embedding parameters with a separate Chinchilla-style...
Load Hijack modifies nothing but router weights in a checkpoint. When a private trigger appears, token-to-expert assignment concentrates on experts co-located on a single GPU, making it a straggler while peers idle (arXiv 2608.10614). Across three MoE families and four corpora...
arXiv 2607.14530 gets Hyper-Connections past the N=4 wall by sparsely updating only k=4 streams plus temporal feature augmentation, scoring 4.0 points higher on average downstream than prior mHC on an 18B MoE. Vanilla and mHC need 1.50x and 1.19x xHC's compute to hit the same...
The authors extract a steering direction from the model's existing tool-use preference signal and apply it at inference, producing monotonic control over how often the agent reaches for a tool while keeping invocations valid (arXiv 2608.25198). Open-domain QA accuracy with liv...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.