SP3O names 'Value Flattening' as an overlooked PPO failure and fixes it by supervising only three states per response

arXiv / HuggingFace Daily Papers·medium signalXBlueskyLinkedInCopy link

arXiv 2609.18708, submitted 2026-09-16, identifies critics whose predictions stay flat while true Monte Carlo state values swing sharply across intermediate states. The authors trace it to an implicit variance penalty in the critic loss plus redundant gradients from temporally adjacent states with near-identical representations. SP3O applies value supervision to a few well-separated states per response instead of all of them, and three supervised states was enough to mitigate the effect and improve the learned policy consistently across Qwen3-Base sizes, with controlled FrozenLake runs showing the problem worsens as state space complexity grows.