ATOBench Probes What Pentest Agents Do When the Target Lies to Them

arXiv 2608.12996·medium signalXBlueskyLinkedInCopy link

ATOBench injects registered response transformations at runtime and pairs each transformed episode with a native episode in the same environment, aligned at the first affected response, then reconstructs how the agent handled later actions, evidence recovery, stopping, and report support. Three frozen observation contracts cover exploit proof, resource ownership and reusable artifacts across 450 episodes and five model routes. The headline result for anyone reading agent pentest reports: increased activity can mask a broken verification chain, and successful recovery depends on finding usable evidence and preserving it all the way through reporting — outcome-only evaluation cannot see either.