Linear Probes on an Open-Weight Reviewer Beat the Same Model's Own Verdict at Spotting Vulnerable Code, 61–67% vs Coin-Flip Prompting

arXiv 2608.09643·high signalXBlueskyLinkedInCopy link

Posted 2026-08-10, this paper trains one linear probe per model on paired vulnerable/fixed Python functions, then tests it without retraining on real disclosed CVEs whose weakness type the probe never saw. Across five open-weight reviewer models the probe ranks the vulnerable function above its fix in 61–67% of single-function fixes, beating both the 50% chance line and the same model's prompted YES/NO logit win-rate under every prompt tried. Asking the model for a written verdict — even with chain-of-thought — returns the same answer on vulnerable and fixed code most of the time. For teams reviewing closed-weight agents' output, activations carry a signal prompting throws away.