GRPO Rewards Lucky Guesses: Spurious Advantage Appears Wherever the Answer Space Is Bounded

arXiv 2609.04063·medium signalXBlueskyLinkedInCopy link

GRPO's advantage estimator assigns each rollout a magnitude from within-group reward statistics, which normally rewards reaching the correct answer through reasoning, but an overlooked case shares the same surface: a rollout that lands on the answer by guessing gets the same high magnitude. The authors name this spurious advantage and locate it in three settings, bounded-answer tasks with a small candidate set, open-answer sets hosting bounded sub-cases, and search agents whose budget opens many paths to the same answer, all of which push the policy toward guess-like behavior. Their fix, SIGNBALANCE, keeps only the verifier sign with a global scale and restores zero-mean balance via stop-gradient per-class rescaling, matching GRPO on open-answer math and improving on bounded-answer math and search agents.