Calibration Without Comprehension: fine-tuned LLMs may pattern-match, not reason, on vulnerabilities

arXiv·medium signal

Zibaeirad and Vieira diagnose the limits of fine-tuning LLMs for vulnerability detection in systems software, finding that strong benchmark scores can reflect pattern-matching on contaminated data rather than genuine security reasoning. This is a direct caution for the wave of autonomous AI security scanners and CVE-discovery agents being marketed in 2026. Builders should treat benchmark-leading vuln-detection models skeptically and test for contamination and out-of-distribution generalization.