Skip to content

Wire Briefings Search

Fetching from the wire…

01

02

03

04

05

06

07

08

Wire Briefings Search

Entity trail

ORCA-bench

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

0

Findings

1

Edges

1

Sources

1

Corpus findings

2026-07-31 / arxiv-researcherORCA-bench: Best Frontier Agent Scores 25.3% on Realistic Oncall Root-Cause Analysis, 10.0% on Hard TasksORCA-bench (arXiv 2607.28545, July 30) pairs a live OpenTelemetry-instrumented microservice system — six days of metrics, logs, and traces exposed via Prometheus, Jaeger, and OpenSearch through Grafana, plus full source access — with 1,079 root-cause-analysis tasks that vary report specificity, time-to-detection, and co-occurring faults. Across five frontier agents the best RCA accuracy is 25.3% on Medium (realistic-input) tasks and 10.0% on Hard, a gap that persists even with Claude Fable 5; the weakest model hallucinates an implausible root cause in 40% of reports, and removing source-code access degrades every metric. Ground truth was signed off by expert SREs with LLM-as-judge re-scored by humans at Cohen's weighted kappa 0.90, and the authors frame the 50 GB/six-day testbed as a lower bound on real production difficulty.

Graph relationships

BENCHMARKED AGAINST
Claude Fable 5
Claude Fable 5 tested on ORCA-bench achieves low accuracy on hard root-cause analysis tasks
Source finding

Source trail

Graph sources

entity graphfindings textkg edgeskg entities