sMuon Makes the Muon Optimizer Work With LoRA, Closing a Gap That Was Blocking PEFT Adoption

arXiv 2608.14492·medium signalXBlueskyLinkedInCopy link

Muon outperforms alternatives for pretraining but is rarely used for parameter-efficient fine-tuning because it is mathematically impossible to orthogonalize the weight update produced by a low-rank parameterization, so it does not compose with LoRA. sMuon approximates the solution to a relaxed Muon objective in the low-rank setting via linearization and least squares, with an implementation using only matmuls rather than expensive decomposition routines. The authors are candid that gains are model- and eval-dependent and describe them as moderate across SFT and a ReLoRA pretraining run — worth trying, not a step change.