Optimizer-Model Consistency: Matching Pre-training Optimizer During Fine-Tuning Reduces Forgetting

arXiv·medium signal

Full finetuning with the same optimizer used during pretraining achieves a better learning-forgetting tradeoff than other optimizers and, surprisingly, LoRA. The phenomenon is termed 'optimizer-model consistency.' This challenges the widespread assumption that LoRA always preserves more pre-training knowledge than full finetuning — the optimizer choice during SFT may matter more than the method itself.