Replay Selection Alone Matches Full Replay Pipelines and Cuts Agentic-Search Wall-Clock Time

arXiv 2609.03941·medium signalXBlueskyLinkedInCopy link

RL post-training is bottlenecked by fresh rollout generation, and in agentic settings environment interaction dominates wall-clock cost, but existing replay methods bury their contribution inside exploration and mixed-policy machinery. Headroom-Drift Replay is a group-level control primitive for GRPO splitting reuse into two decisions: Headroom ranks stored groups by remaining learning value, Drift gates them by compatibility with the current policy, with the fresh on-policy stream untouched and no auxiliary generation added. Across mathematical reasoning, multimodal reasoning and Agentic Search it beats naive replay and matches or exceeds broader replay methods on Avg Mean@32, delivering comparable quality at materially lower wall-clock time where environment calls dominate.