Adam Isn't Gauge-Equivariant, So It Can't Inherit Gradient Descent's Low-Rank Bias — 43-44% Held-Out Error Gap at Matched Loss
On a factored model W = UV-transpose, gradient descent is implicitly biased toward low-rank solutions and Adam is not; the paper traces this to gauge symmetry, proving gauge-equivariance is necessary (though not sufficient) for the transfer. Gradient descent, momentum, shared-scalar Adam, Muon, and Shampoo satisfy it; Adam, RMSProp, and other coordinate-wise methods do not, and a one-parameter family sweeping from coordinate-wise to shared-scalar preconditioning restores the bias monotonically, isolating anisotropy as the cause. In transformers Adam separates two gauge-equivalent initializations at the first step and ends with per-head WQ-transpose-WK invariants 56% apart in relative Frobenius distance, and on two hyperspectral datasets at matched training loss gradient descent cuts held-out error 43-44% at the lowest sampling density — directly relevant to anyone tuning low-rank adapters.
↳ Follow the thread