Pion: Spectrum-Preserving Optimizer for LLM Training via Orthogonal Transformation

arXiv·medium signal

Kexuan Shi et al. introduce Pion, a new optimizer for LLM training that preserves the spectrum of weight matrices through orthogonal equivalence transformation. Unlike Adam and its variants, Pion maintains spectral properties during optimization, which the authors show leads to more stable training dynamics. Targets a known failure mode in large-scale training where spectral collapse degrades model quality.