Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-28 · high
CTF-ABACUS traced 1,435 agent attempts across 240 challenges and found only 62-87% of flags backed by demonstrated exploitation.
Why now: The study posted August 26 and covers 1,435 attempts across 240 challenges.
CTF-ABACUS, a new benchmark, checked whether AI agents that win capture-the-flag hacking challenges exploited anything, and many didn't.
That gap matters for anyone using a single pass rate to judge agent security research. Depending on the model and which judge scores it, up to 38% of flags counted as wins showed no evidence the agent broke in.
The study breaks down each agent run into penetration-testing phases and techniques, pinpointing where exploitation happened and where the flag first appeared. Researchers ran two separate judging methods across 1,435 attempts by six frontier and open-source models on 240 challenges.
Untraced flags came from four places: exposure the agent could simply read, memorized training recall, external lookup, or guessing. Shortcut recoveries also follow shallower trajectories than verified exploits, doing visibly less work along the way. That makes them detectable if anyone inspects the trace instead of the flag alone.
The paper doesn't say which of the six models leans hardest on shortcuts, or whether the 62-87% range holds steady across challenge difficulty.
Each link below shares sources, entities, or timing with this story.
Same source domain / Semantically similar
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.75).
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.76).
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.75).
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.75).
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.75).
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.74).
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.73).
Reported by the same outlet (arxiv.org); covers closely related ground (similarity 0.73).