Research2026-08-27 · source-backed

CodeRabbit blocks malicious PRs but names the target vulnerability 16 of 31 times, and 0 of 7 when evidence sits outside touched files.

Issue

2026-08-27

Confidence

source-backed

Sources

Redaction

redacted

Story

MalPR-Bench covers 89 malicious pull requests plus 50 benign controls across 44 repositories and eight language families, each with a pre-committed rubric giving no credit for off-target findings (arXiv 2608.25730). The authors name the Verdict-Diagnosis gap: a reviewer can block a PR for an unrelated issue, and fixing what it reported leaves the real defect exploitable. On 31 held-out malicious PRs, PRGuard and CodeRabbit block comparably at 22 and 24, but PRGuard identifies 22 target vulnerabilities to CodeRabbit's 16. On 14 absence-type cases both block 9 while PRGuard identifies 9 targets to CodeRabbit's 3.

Source trail

arXiv 2608.25730 More from arxiv.org

Entities

Bench CodeRabbit Diagnosis MalPR PRGuard PRs Verdict

Provenance

Canonical issue: Ramsay Research Agent — August 27, 2026
AI generated: no
Story unit: 2026-08-27-coderabbit-blocks-malicious-prs-but-names-the-target-vulnerability-16-of-31-times-and-0-of-7-whe
Labels: source-backed, canonical briefing excerpt

Fetching from the wire…

CodeRabbit blocks malicious PRs but names the target vulnerability 16 of 31 times, and 0 of 7 when evidence sits outside touched files.

Story

Related stories

Three Studies, Same Answer: AI Coding Is Net-Negative After Rework

CodeRabbit raised $143M at a $1.5B valuation to govern AI-written code: 2M reviews a week, 17,000 customers, revenue up more than 5x.

Skill-only evolution beat memory pairing in Claude Code

Agents' Last Exam: frontier agents pass 2.6% of the hardest economically-valuable tasks.

66.5% of Malicious GitHub Issues Get Past Every Guardrail in Cursor, Claude Code, and Codex

First Large-Scale Study of AI Coding Bots in CI/CD.

AI Coding Agents Write Vulnerable Code 87% of the Time. Every Agent. Every Test.

Codex on GPT-5.5 tops Terminal-Bench 2.1 at 83.4%, but the top Claude config is export-banned.

Source trail

Entities

Provenance

CodeRabbit blocks malicious PRs but names the target vulnerability 16 of 31 times, and 0 of 7 when evidence sits outside touched files.

Story

Related stories

Three Studies, Same Answer: AI Coding Is Net-Negative After Rework

CodeRabbit raised $143M at a $1.5B valuation to govern AI-written code: 2M reviews a week, 17,000 customers, revenue up more than 5x.

Skill-only evolution beat memory pairing in Claude Code

Agents' Last Exam: frontier agents pass 2.6% of the hardest economically-valuable tasks.

66.5% of Malicious GitHub Issues Get Past Every Guardrail in Cursor, Claude Code, and Codex

First Large-Scale Study of AI Coding Bots in CI/CD.

AI Coding Agents Write Vulnerable Code 87% of the Time. Every Agent. Every Test.

Codex on GPT-5.5 tops Terminal-Bench 2.1 at 83.4%, but the top Claude config is export-banned.

Source trail

Entities

Provenance