SAPO Shares One Autoregressive Backbone for Policy and Value, Beating PPO by 15.1 Points With No Separate Critic

arXiv 2608.19842·high signalXBlueskyLinkedInCopy link

Group-relative agentic RL methods like GRPO avoid PPO's critic memory cost but lack temporal credit assignment and suffer advantage collapse on long-horizon tasks. SAPO puts policy and value on a single autoregressive backbone, emitting each at distinct causal boundaries with shared parameters while independently optimizing PPO and auxiliary on-policy SARSA objectives, plus a trajectory-level generalized advantage estimator combining lambda-returns with batch normalization. On ALFWorld and WebShop with Qwen2.5-1.5B/7B, SAPO beats PPO by a mean 15.1 percentage points and GRPO by 12.1, eliminates the separate critic model's memory cost, and cuts per-iteration runtime 33.2% versus PPO.