37,623 provenance-labeled agent PRs: Codex code was reverted half as often as human code, Devin's 31% more, and Claude Code PRs waited 12.6 hours for first review
A study of 37,623 PRs from OpenAI Codex, Devin, GitHub Copilot, Cursor and Claude Code across 2,807 GitHub repositories (Dec 2024 to Jul 2025), combining the AIDev dataset with 58,792 cached GitHub API responses, finds quality is vendor-specific rather than a property of 'AI code'. Codex PRs were reverted 6.1% of the time against a human baseline of 11.5% (OR 0.50) while Devin hit 14.5% (OR 1.31), and agent code pooled across vendors was less likely than human code to carry a security smell (OR 0.63), driven by fewer hardcoded credentials and eval-style constructs. Review effort concentrated unevenly: Copilot PRs drew the most human reviews and change requests, Claude Code PRs waited longest for a first human review at a 12.6 hour median. For a builder this says pick the agent by measured post-merge revert rate on your own repo rather than by benchmark score, and budget review latency separately from generation quality.
↳ Follow the thread