RealReplicaBench Makes Agents Actually Change State in Cloned Alibaba, Shopify and FreightOS Services — Best Score Is 61.7%

GitHub·high signalXBlueskyLinkedInCopy link

Accio open-sourced RealReplicaBench on August 2 (1,042 stars by August 9) with 107 tasks across 14 containerized mock services replicating real commerce and logistics platforms including Alibaba, Shopify, and FreightOS. The task mix is 53 CLI, 28 browser, 16 file operations, and 10 API/MCP, split into 65 text-only, 20 browser-text, and 22 vision-requiring; twelve model families were run on two harnesses. Claude Opus 5 led both, scoring 66/107 (61.7%) on the Accio harness and 60/107 (56.1%) on OpenClaw — meaning the best available agent still fails roughly four in ten real business workflows that require mutating state rather than answering questions.