CTF-ABACUS finds only 62-87% of agent CTF flags are backed by a trace showing actual exploitation

arXiv·high signalXBlueskyLinkedInCopy link

Posted 26 August, CTF-ABACUS reconstructs each agent run as an evidence-grounded solve profile, decomposing actions into penetration-testing phases and techniques to identify where exploitation happened, where the flag first appeared, and whether the recovered flag is supported by demonstrated behavior. Applied to 1,435 attempts by six frontier and open-source models across 240 challenges under two judge lenses, trace-verified exploits account for only 62-87% of recovered flags, with the remainder coming from direct flag exposure, memorized recall, external lookup or guessing. Shortcut recoveries follow substantially shallower trajectories, so binary CTF pass rates overstate offensive security capability.