Debug computer-use agents by diagnosing suspect steps, not replaying whole trajectories: re-execution success more than doubles, 12.2%→25.86%

arXiv 2608.02643·medium signalXBlueskyLinkedInCopy link

CUADebug hand-annotates 204 failed OSWorld trajectories and finds the failure distribution is not where most tooling looks: task reasoning and control accounts for 110 of 204 failures, versus 36 for perception, 25 for grounding/interaction, and 13 for external/system errors. Rather than feeding an entire trajectory to a model, CUADebugger examines questionable steps with paired screenshots and traces to emit a structured diagnosis — root cause, error class, evidence, corrective strategy — raising joint subtype-and-step diagnosis from 11.2% to 19.6% on Gemini 2.5 Pro. That diagnosis pays off downstream: single re-execution reaches 29.90% against 13.89% for history-only continuation, and continual re-execution climbs from 12.2% to 25.86%, approaching the 29.21% human-oracle ceiling.