Fetching from the wire…
01
02
03
04
05
06
07
08
Agents2026-07-26 · source-backed
Chen et al. extend AppWorld into a 516-task user-in-the-loop benchmark across nine simulated apps, injecting ambiguities and constraints that force the agent to ask clarifying questions, request confirmation, or declare a task infeasible (arXiv 2607.20536). Opus 4.7 gets 48.6% overall, 35.7% compositional, 21.3% under the stricter scenario-level metric. User behavior is simulated by an LLM with designed knowledge boundaries rather than the unconstrained simulators prior work used. The analysis finds correct interaction, not tool execution, determines success, which is a different failure axis than most agent evals measure.
Each link below shares sources, entities, or timing with this story.
Opus built by Anthropic / Shared entities / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Opus built by Anthropic); both cover Claude Opus, LLM, Opus; overlapping topics (benchmark, opus).
Simon Willison released LLM / Shared entities / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover Claude Opus, Opus; overlapping topics (opus, task).
Opus built by Anthropic / Shared entities / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Opus built by Anthropic); both cover Claude Opus, Opus; overlapping topics (agent, benchmark).
Linked by a graph relationship (Opus built by Anthropic); both cover Claude Opus, Opus; overlapping topics (agent, opus).
Claude Code uses Opus / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Claude Code uses Opus); both cover Claude Opus, Opus; overlapping topics (agent, opus).
Opus built by Anthropic / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Opus built by Anthropic); both cover LLM, Opus; overlapping topics (agent, task).
Claude Code uses Opus / Shared entity: Opus / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Claude Code uses Opus); both cover Opus; overlapping topics (benchmark, opus, task).
Claude Code uses Opus / Shared entity: LLM / Same source domain / Shared topic / Earlier coverage
Linked by a graph relationship (Claude Code uses Opus); both cover LLM; reported by the same outlet (arxiv.org).