Complete-muE: Hyperparameter Transfer Framework Bridges Dense FFN and MoE Transformer Training

arXiv·medium signal

Complete-muE enables hyperparameter transfer across dense feed-forward network and any Mixture-of-Experts configuration in transformers. Practitioners can tune hyperparameters on a small dense model and transfer them to large MoE setups, dramatically reducing the cost of hyperparameter search. Directly addresses one of the most expensive practical problems in training MoE architectures like Mixtral or DeepSeek variants.