AppWorld-UL: Claude Opus 4.7 hits 48.6% when the agent has to actually talk to the user.

Chen et al. extend AppWorld into a 516-task user-in-the-loop benchmark across nine simulated apps, injecting ambiguities and constraints that force the agent to ask clarifying questions, request confirmation, or declare a task infeasible (arXiv 2607.20536). Opus 4.7 gets 48.6% overall, 35.7% compositional, 21.3% under the stricter scenario-level metric. User behavior is simulated by an LLM with designed knowledge boundaries rather than the unconstrained simulators prior work used. The analysis finds correct interaction, not tool execution, determines success, which is a different failure axis than most agent evals measure.