Best Practice Critic Optimization Matches GRPO While Sampling One Response Per Prompt

arXiv 2608.23566·medium signalXBlueskyLinkedInCopy link

GRPO-style group methods avoid training a critic by sampling many responses per prompt, which is expensive; critic-based training is cheaper but historically unstable. BPCO is a recipe combining DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages and length-adaptive GAE, with controlled experiments isolating each choice. Because the critic only exists at training time it can be conditioned on information hidden from the policy, such as a reference answer or grading rubric. Across mathematical reasoning from 1.5B to 30B-A3B MoE models, BPCO improves a strong critic-based baseline consistently and matches or exceeds a group-based baseline at one sample per prompt. Code at github.com/QPHutu/golden_critic.