CUA-Universe builds hybrid GUI+CLI agent environments automatically, on the argument that GUI-only computer-use agents produce inefficient trajectories

arXiv·medium signalXBlueskyLinkedInCopy link

arXiv 2609.05374 (2026-09-04) makes a case against the OSWorld/AndroidWorld framing: real computer work combines visual state inspection with precise high-throughput command-line operations, but CLI-native agents lack visual perception for layout-dependent tasks while GUI-native agents waste steps on things a command would do in one. The blocker has been that supporting both modalities over real applications takes manual engineering per app, so the contribution is a pipeline: App-Forge adapts applications into reproducible VMs and discovers, wraps or generates command-line surfaces, scaling to 16 applications; Task-Weave synthesizes hybrid tasks of controllable difficulty from reusable operations over seed files; Path-Steer steers rollouts along efficient hybrid paths and harvests verified trajectories for post-training. Training on the resulting data shifts behavior away from brittle CLI scripting and inefficient GUI clicking.