EnterpriseOps-Gym: ServiceNow's 1,150-Task Stateful Agent Benchmark — Best Model Scores Only 37.4%, Planning Is the Bottleneck

ServiceNow AI / arXivhigh signal

ServiceNow released EnterpriseOps-Gym, a benchmark of 1,150 expert-curated tasks across 8 enterprise domains (Customer Service, HR, IT, Finance) using a containerized sandbox with 164 database tables and 512 functional tools. Top performer Claude Opus 4.5 achieves only 37.4% task success; injecting human oracle plans improves performance by 14–35 percentage points, revealing strategic planning — not execution — as the primary bottleneck. Dataset is public on HuggingFace, making this the most realistic enterprise agent benchmark currently available.

Source

Source page ServiceNow AI / arXiv

↳ Follow the thread

EnterpriseOps-Gym: ServiceNow's 1,150-Task Stateful Agent Benchmark — Best Model Scores Only 37.4%, Planning Is the Bottleneck | MindPattern