20% of AI-Generated Vulnerability Patches That Pass CI Break Production — SCA RemBench Benchmark of Claude Code, Gemini 3 Pro, and Codex

Backline.ai / Hacker News·medium signal

Backline.ai benchmarked Claude Code (79.3/100), Gemini 3 Pro (82.1/100), and Codex (76.3/100) on 25 real-world repos with verified CVEs using the SCA RemBench framework. Result: 20% of AI patches that pass CI tests break production functionality. Specific failure modes include Express upgrade where Gemini stubbed removed utilities with null returns (silently breaking query parsing), and urllib3 patches that actively hid newly available attributes. Their key finding: 'Vulnerability remediation isn't a coding problem — it's a planning problem that ends with coding.' Adding structured planning phases improved scores to 89.5 (+7.4 points).