ActObs: supervising environment observations during SFT changes how agents explore under RL, +3.4pp pass@16 on Terminal-Bench 2.0

arXiv / HuggingFace Daily Papers·medium signalXBlueskyLinkedInCopy link

arXiv 2609.20715 (submitted 2026-09-17) challenges the convention that supervised fine-tuning should apply loss only to agent-authored action tokens while using environment observations as mere context. ActObs also supervises the observation tokens already present in every trajectory, adding no data, parameters, sequence tokens or forward passes. The two approaches look identical after SFT and diverge after GRPO: on Qwen3-4B, ActObs beats action-only at every sampling budget on Terminal-Bench 2.0, and on Qwen3-8B it trades some pass@1 for +3.4pp at pass@16 while solving more distinct tasks, with a +4.2pp pass@1 gain at 4B on unseen aider-polyglot code editing. The mechanism is that action and observation gradients become orthogonal quickly during SFT, leaving action-only training with a large unexploited residual.