Validating a security patch by re-running the crash PoC inflates agent solve rates 1.83x
arXiv 2609.04075 shows two validity failures in C/C++ vulnerability-patching evaluation: 25% of agent patches are substantially similar to the historical developer patch (memorization), and agents frequently patch on the crash stack trace to suppress the reported crash rather than fixing the root cause. PatchBench counters this by selecting vulnerabilities whose ground-truth fix lies outside the crash stack and transplanting historical vulnerabilities into mutated repository contexts; across 11 agents including the top three AIxCC entrants, PoC-only validation inflated the solve rate by 1.83x on average. If you let an agent fix a crash, require the fix to land outside the frames in the stack trace before you call it done.
↳ Follow the thread