You can predict a web agent's failure mid-run from observable trajectory signals alone, no logits required

arXiv 2609.02057·medium signalXBlueskyLinkedInCopy link

arXiv 2609.02057 (2026-09-02) studies prefix-level risk prediction when model-internal uncertainty is unavailable, which is the situation for anyone driving a hosted model. It derives Macro features summarizing cross-step agent-environment behavior and feedback, and Micro features measuring consistency of intention, action, and anticipated state change via repeated black-box queries. Crucially it does not inherit the final result label: it labels the first uncorrected critical error associated with final failure as the key-step boundary, so early prefixes of failed runs still count as on track. Across WebArena-Lite and Online Mind2Web with five open and closed backbones, observable signals were competitive with internal-signal baselines, supported early intervention under fixed false-cut budgets, and transferred across held-out website categories.