From Controlled to the Wild: Evaluating AI Pentesting Agents on Real-World Targets

arXiv·medium signal

This paper identifies the gap between current CTF-style benchmarks and real-world offensive security, showing that AI pentesting agents optimized for predefined goals (flag capture, exploit reproduction) fail to transfer to open-ended real-world targets. The authors propose evaluation protocols that capture the complexity and ambiguity of actual penetration testing, finding that current agents achieve less than 15% of the expected coverage on production-like environments.