CodeRabbit Blocks Malicious PRs but Names the Actual Vulnerability Only 16 of 31 Times, and 0 of 7 When Evidence Sits Outside Touched Files

arXiv 2608.25730·high signalXBlueskyLinkedInCopy link

Chen et al. build MalPR-Bench, 89 malicious pull requests plus 50 benign controls across 44 repositories and eight language families, each with a pre-committed rubric that gives no credit for off-target findings. They name the Verdict-Diagnosis gap: a reviewer can block a PR for an unrelated issue, and fixing what it reported leaves the real defect exploitable. On 31 held-out malicious PRs, their PRGuard and CodeRabbit block comparably (22 vs 24) but PRGuard identifies 22 target vulnerabilities against CodeRabbit's 16, and on 14 absence-type cases both block 9 while PRGuard identifies 9 targets to CodeRabbit's 3.