VLoc Bench: the best of 27 models scores 0.229 File F1, and 38.4% of tasks get nothing correct from anything.

Agents get only a CWE description and terminal access, and have to find the implementing files across 500 real vulnerabilities from 290 repositories, six package ecosystems and 147 CWE categories (arXiv 2609.15939). Across 27 language models and four static-analysis tools on a standardized interface, the strongest system reached File F1 of 0.229, and 38.4% of tasks received no correct localization from any evaluated system. The authors also found systems that identify vulnerabilities well still report unsupported locations on already-patched repos, which separates localization from detection as a distinct capability. Anyone selling agentic vulnerability discovery should be asked about this number.