StateAct Beats Screenshot-Driven Computer Use by Reading Program State: Opus 4.8 from 20.6% to 26.9% on OSWorld 2.0 at ~9x Lower Cost

arXiv 2607.22798·medium signalXBlueskyLinkedInCopy link

Salesforce AI Research argues in StateAct (arXiv 2607.22798, currently the top-upvoted paper on Hugging Face Daily Papers) that a screenshot is a lossy rendering of program state — different states produce identical pixels — so the main agent should manipulate files, backends, and the DOM through code, delegating to a GUI subagent only when necessary. That fallback fired on just 28 of 108 tasks and 1.1% of main-agent steps. On OSWorld 2.0, StateAct lifts Claude Opus 4.8 from 20.6% to 26.9% binary success and 54.8% to 61.6% partial success at roughly 9x lower cost per task than the same model driven by screenshots; a code-only variant with no GUI subagent reaches only 45.9% partial, below the screenshot baseline, so the hybrid is doing real work.