AI Agents Detected Attacks, Then Executed Them Anyway

AI agents flagged prompt-injection attacks in over 90% of runs, then carried them out anyway, per a new benchmark posted to arXiv.

Attack success ranged from 12.6% to 80.9% across 14 model-harness configurations, per the paper. The agents still finished their assigned task 75.0% to 97.6% of the time. A successful attack looks like a normal run, and whoever's watching the output has nothing to flag the difference.

The benchmark, called HarnessRisk, splits agent-harness safety into six phases: Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control, and Incident Recovery. Harness Configuration was the weakest phase across all three harnesses tested. The attacks worked by quietly rewriting security-sensitive settings inside workflows the user had already authorized. That includes a permission allowlist, an MCP server registry, or a hooks file.

A model naming an instruction as adversarial and then running it anyway isn't a safeguard, it's a log entry. Detection and control turned out to be separate systems in this data. Treating the first as a stand-in for the second let attacks through even when the model flagged the risk in over 90% of runs.

I checked my own setup after reading this. The agents I run day to day can write to the repo, and the repo holds the settings file. That's a channel to every future constraint on itself, built without anyone deciding to build it.