Top AI agent finds only 25% of oncall root causes

The best of five frontier agents solved just 25.3% of realistic root-cause tasks on a new oncall benchmark, per ORCA-bench's authors. That's the number SRE and platform teams should plan around before handing an agent the pager. On the Hard tier, the top score drops to 10%. The weakest model invents an implausible root cause in 4 of every 10 reports.

ORCA-bench runs on a live, OpenTelemetry-instrumented microservice system. It provides six days of real metrics, logs and traces from Prometheus, Jaeger and OpenSearch, plus full access to the source code. Agents work through 1,079 tasks that vary how specific the incident report is and whether more than one fault is firing at once. Ground truth came from expert SREs, with agreement scored at a Cohen's weighted kappa of 0.90, so the scoring isn't the weak point.

Claude Fable 5 was one of the five agents tested, and the same gap shows up there too. Removing source-code access dropped every model's score, across every metric the benchmark tracks.

That access dependency is the real story here. These agents aren't reasoning their way to a root cause, they're using the codebase to narrow down what to blame. Watch whether oncall-agent pitches start listing full repo access as a requirement in the fine print.