Sequential Fine-Tuning Drops Models 23.1% Below Baseline

Sequential fine-tuning drags models 23.1% below their base performance while sequential reinforcement learning pushes them 24.9% above it, per a Tsinghua-led paper posted on Hugging Face.

The gap matters for anyone chaining task-specific fine-tuning runs across math, coding, logic, and science on one model. Standard sequential fine-tuning can leave it worse than where it started.

The paper ties the split to how each method moves parameters. RL updates run roughly 100x smaller in magnitude than SFT updates. Gradients across different RL tasks share about 10⁻⁵ similarity in parameter space, versus roughly 10⁻¹ for SFT tasks.

That gap means RL tasks barely interfere with each other during training, while SFT tasks overwrite each other's progress as they're chained.

The paper's fix trains a separate model per task with RL, then merges the results, in a recipe it calls Parallel-RL. That approach keeps 102.8% of the performance a single-task RL model would hit on its own, using only 5% additional adaptation to merge.

That mechanism should hold outside this paper's four benchmarked domains, not just within them. Worth checking whether the 100x gap in update magnitude survives on models larger than the ones tested here.