Adam isn't gauge-equivariant, so it can't inherit gradient descent's low-rank bias. 43-44% held-out error gap.

arXiv 2608.05136 proves gauge-equivariance is necessary (not sufficient) for the transfer on a factored model W = UVᵀ. GD, momentum, shared-scalar Adam, Muon and Shampoo satisfy it. Adam, RMSProp and other coordinate-wise methods don't. A one-parameter family sweeping from coordinate-wise to shared-scalar preconditioning restores the bias monotonically, isolating anisotropy as the cause. In transformers Adam separates two gauge-equivalent initializations at the first step, ending with per-head WQᵀWK invariants 56% apart. Directly relevant if you're tuning LoRA adapters.