Le Critique Revives Value Functions for LLM RL With Privileged Signal and an Adaptive GRPO Interpolator

arXiv 2608.16739·medium signalXBlueskyLinkedInCopy link

Critic-free methods like GRPO won on practicality, but they give only sequence-level credit and stall on straggler rollouts, increasing off-policyness. This work proposes Privileged Value Functions, which inject extra task-relevant token-level signal without biasing the policy objective, plus TETHER, a baseline that adaptively interpolates between group-relative and value baselines according to how accurate the value function currently is. Across several reasoning tasks both consistently improve on the standard value-function baseline and are competitive with or better than mean-baseline GRPO — a credible case for putting critics back in the pipeline.