Anthropic turns its CI on-call agent into a public setup kit

Anthropic published the setup kit behind Claude Tag, the agent that triages its own CI/CD failures, at github.com/anthropics/oncall-kit. Internally, Tag runs under a dedicated service account with MCP connectors into Datadog, Grafana, PagerDuty, GitHub and Kubernetes, and an orchestrator that spawns executor subagents to check dependencies in parallel. Median time to first analysis is 14 minutes, per the August 18 writeup.

The part worth studying isn't the connector list. It's where the standing instructions live: markdown skills checked into a GitHub repo, including an oncall.md with deterministic escalation rules and a lessons.md that updates itself as incidents happen. That's a plain-text incident memory a team can read, diff, and argue with in a pull request, instead of a black box that relearns nothing between runs.

The public kit strips this down to reference code, 21 stars, not maintained, and structures the setup as five gated phases. Phase one mines your own incident history to build playbooks. Before anything gets installed, those playbooks get validated against held-out incidents, the ones not used to write them. Claude stays strictly read-only throughout.

That's the opposite of the eager, semi-autonomous agents most on-call tooling has been selling. It won't touch your infrastructure until it's proven it can predict outcomes on incidents it hasn't seen. A related thread of senior engineers describing how they actually review Claude Code output, not just accept it, points at the same instinct: verification before trust, every time, not once at demo day.

What it doesn't say

The writeup doesn't say how oncall.md's escalation criteria were tuned, or what happens when the held-out validation fails. If you're building something similar, that gap is where the real work sits.