APPO: Agentic Procedural Policy Optimization for multi-turn tool use

arXiv·high signalXBlueskyLinkedInCopy link

APPO is a new agentic reinforcement learning method aimed at improving the multi-turn tool-use capabilities of LLM agents, where recent agentic RL advances have shown large gains but remain unstable across long tool-calling trajectories. The paper (arXiv 2606.12384, Wang et al.) frames policy optimization at the procedure level rather than per-token, which matters for builders fighting reward sparsity and credit assignment in long agent rollouts. Worth watching as multi-turn tool-use RL becomes the standard recipe for coding and research agents.