Minimal AI Scaffold Rediscovers 68% of Known CVEs

Researchers built HoF-Bench from 95 real CVEs AISLE's analyzer found, then rediscovered 68% without a single frontier model, per an arXiv paper posted July 29.

For teams weighing whether vulnerability hunting needs a frontier model, that's the real number to sit with. Sixty-five of 95 real CVEs came back without one, and the scaffold around the model decided what counted as a find, not the model's size.

The benchmark pins each vulnerability at its exact vulnerable commit across eight repositories where AISLE's analyzer originally flagged it. A judge model blind to which detector produced which answer only credits a match. It has to name the right code path, root cause, attack condition, and impact, per the paper.

Ten models did the detecting across 7,600 model-CVE passes. Five are open-weight, from 21 billion to 284 billion total parameters with 3 to 13 billion active. Five are proprietary models at the small or flash tier. None is a frontier model.

Yes, but the gap isn't spread evenly. Every CVE all ten models missed sits in C infrastructure code, per the paper.

A related benchmark found autonomous pentest agents landing zero real-target exploits against a decoy environment. A separate piece argues for checking a model's tool-call claims against server-held facts instead of trusting its own rationale. Same lesson twice: verify the finding, don't take the model's word for it.