First Large-Scale Study of AI Coding Bots in GitHub Actions CI/CD Finds Substantial Failure Rate Gaps

arXiv·high signal

Researchers analyzed 61,837 GitHub Actions workflow runs from 2,355 repositories triggered by PRs from five AI bots: Claude, Devin, Cursor, Copilot, and Codex. The study found substantial differences in CI/CD pass rates across bots, revealing that agentic AI code contributions have measurably different reliability profiles in automated pipelines. This is the first empirical study quantifying how AI-generated code performs under real CI/CD constraints at scale.