3PO Explores in Parameter Space Instead of Temperature, Beating GRPO at Near-Identical FLOPs on OLMo-3 7B and Qwen2.5-Math-7B

arXiv 2608.09805·medium signalXBlueskyLinkedInCopy link

Published 2026-08-10, this work argues action-space exploration levers like temperature scaling can only reshape output variance, never reorder tokens — limiting exploration and causing divergence or stalled RLVR training. Perturbed Parameter Policy Optimization (3PO) instead samples different policies from a posterior so each rollout comes from a distinct policy, making policy diversity a complementary control lever. On OLMo-3-1025-7B and Qwen2.5-Math-7B across math reasoning and code generation, 3PO variants consistently beat standard GRPO at near-identical FLOPs cost, and multiple parameter samples produce fewer zero-advantage groups and fewer malformed rollouts during training.