Pass@k vs Pass^k: Two Agent Eval Metrics That Tell Opposite Stories Above k=3
Anthropic Engineering·high signal
Anthropic's engineering blog formalizes a critical divergence: with 75% per-trial success, pass@3 reaches 98% while pass^k falls to 42%, making them incompatible proxies for reliability. Pass@k suits dev-time exploration (did it ever work?), while pass^k is the correct metric for user-facing agents (does it always work?). At k=10, pass@k approaches 100% while pass^k collapses to near zero — hiding catastrophic reliability gaps from developers who use only the first metric.