Too Correct to Learn: CUTS Decoding Strategy Fixes RL Mode Collapse When Strong Models Saturate Benchmarks

arXiv·medium signal

When strong base models saturate standard benchmarks like MATH, RL training with GRPO collapses because the advantage signal vanishes — all samples are correct but homogeneous. CUTS (Constrained Uniform Top-K Sampling) is a parameter-free decoding strategy that enforces structure-preserving diversity, preventing mode collapse without requiring harder training data. Directly relevant to anyone fine-tuning strong open models with RL on reasoning tasks.