Report pass^k, not just pass@k, or you'll keep shipping flaky agents

Here's the single most useful number I've read this week, and it explains nearly every "great demo, broken in production" story I've ever lived.

pass@k measures best-case: did at least one of k attempts succeed. pass^k measures consistency: did all k attempts succeed. For an agent that's 70% reliable per trial, pass@3 works out to about 97%, while pass^3 is about 34%. That's a 63-point gap between the number you show in the demo and the number your users actually experience (Digital Applied).

Sit with that. A 70%-per-step agent looks like a 97% agent if you measure it the optimistic way and run it three times. The same agent, asked to do three things in a row reliably, succeeds end-to-end about a third of the time. Most agent demos are implicitly pass@k: the author ran it a few times, picked the good run, recorded that. Production is pass^k whether you measure it or not, because real workflows chain steps and users don't get to retry until it works.

This is the metric I wish I'd had two years ago. Every flaky-agent debugging session I've burned a weekend on was a pass^k problem wearing a pass@k disguise. The fix isn't a smarter model, it's gating releases on the consistency number. Decide the chain length that matters for your product, measure pass^k at that length, and don't ship until it clears your bar.

What to do this week: pick your most important multi-step agent flow, run it 10+ times, and compute both numbers. If your pass^k is scary low, that's not a surprise to discover after launch, it's the actual state of your product. Better to see it now. The eval-pipeline skills further down build on exactly this foundation, because once you're measuring consistency honestly, the next questions are how to calibrate your graders and where to source your test cases.