1,800 Pre-Training Runs Show Optimal Learning Rate and Batch Size Shift With MoE Activation Ratio

arXiv 2609.08690·medium signalXBlueskyLinkedInCopy link

Conventional hyperparameter scaling laws fail for ultra-sparse mixture-of-experts models because optimal learning rate and batch size vary with activation ratio in a way neither total nor activated parameter count explains. Across 1,800 pre-training runs spanning six activated-parameter scales, models up to 6B total non-embedding parameters, roughly 20 trillion tokens and 200,000 equivalent H800 GPU-hours, optimal batch size follows a power law in training tokens D while optimal learning rate scales with compute C, and activation ratio A enters both as an additional multiplicative power-law factor. On a held-out 12B-total MoE activating only 1/64 of its experts, the predicted hyperparameters land close to the observed optima.