'SFT Conflicts, RL Coexists': Multi-Stage SFT Lands 23.1% BELOW the Base Model While Multi-Stage RL Gains 24.9% — Because RL Updates Are ~100x Smaller

arXiv / HuggingFace Daily Papers (25 upvotes)·medium signalXBlueskyLinkedInCopy link

This Tsinghua-led paper (arXiv 2608.03573) puts numbers on why sequential fine-tuning across math, coding, logic, and science destroys models: multi-stage SFT ends 23.1% below the base model on average, while multi-stage RL ends 24.9% above it. The mechanism is measured, not asserted — RL parameter updates are roughly 100x smaller in magnitude (10⁻² vs 7.4) and cross-task gradient similarity is ~10⁻⁵ for RL versus ~10⁻¹ for SFT, so RL tasks simply don't collide in parameter space. Their Parallel-RL recipe trains task-specific models independently and merges them, retaining 102.8% of single-task RL performance with 5% additional adaptation, which makes modular per-task training a live option instead of a compromise.