NB-LoRA Fine-Tunes Post-RL Reasoning Models in the Null Space of Their Reasoning Activations to Stop SFT From Erasing Reasoning

arXiv·medium signalXBlueskyLinkedInCopy link

Supervised fine-tuning after RL post-training tends to overwrite the reasoning that RL elicited. Fang, Tzou and Valkov found that reasoning activations sit in low-dimensional subspaces whose approximate null spaces can be estimated from a modest number of examples. They reparameterize LoRA updates through that null basis. Across several RL-trained LLMs, NB-LoRA matched standard LoRA on the new task while keeping reasoning accuracy near pre-fine-tuning levels, including on held-out reasoning benchmarks. This applies to anyone domain-tuning an open reasoning model.