NoRA Normalizes the LoRA Down-Projection and Gets Most of the Benefit From Initialization Alone

arXiv 2608.31036·medium signalXBlueskyLinkedInCopy link

Because LoRA zero-initializes its up-projection, early training dynamics are governed almost entirely by the down-projection, and NoRA exploits that by normalizing the down-projection matrices during training. The paper further shows the same normalization applied only at initialization improves plain LoRA without any repeated normalization, which is close to a free change for existing fine-tuning code. Across pretraining, supervised finetuning and reinforcement learning it reports faster convergence, better stability and reduced catastrophic forgetting, with no additional trainable parameters and no inference-time cost.