SA-MRPO: Multi-Reward RL Wastes Gradient Budget on Already-Solved Objectives; Reweighting by Saturation Gains Up to 5% on AIME24

arXiv 2608.16072 (HuggingFace Daily Papers)·medium signalXBlueskyLinkedInCopy link

Standard practice scalarizes a reward vector with fixed weights before group-wise standardization, which lets rollouts with different reward profiles receive identical advantages and keeps optimizing objectives that are already maxed out. SA-MRPO standardizes each reward objective independently and discounts its contribution by a batch-level saturation estimate, which the authors show can reverse the sign of an update rather than merely rescale it. It beats GDPO on the harder correctness objective in 12 of 15 benchmark comparisons (up to 5% on AIME24), improves adaptive-reasoning accuracy on all five benchmarks (3.8% average, 9.2% on AMC23), and adds up to 2.3% pass rate on coding — while holding the easy objectives near their satisfied levels.