A Survey Reframes 2025-2026 Optimizers as Matrix and Layer Design, Not More Adam Variants

arXiv 2608.28557·low signalXBlueskyLinkedInCopy link

arXiv 2608.28557 (2026-08-28, cs.LG/cs.AI) argues neural-network optimization is no longer well described as a succession of Adam variants, because the design space has widened from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation. It organizes recent optimizers and training-optimization methods along those axes. The sharding and low-precision framing is the practical part for anyone choosing an optimizer for a distributed run rather than a single-GPU benchmark.