Agent harness benchmark finds detection doesn't stop attacks

A new benchmark called HarnessRisk hid adversarial instructions inside untrusted workflow artifacts, things like config files and task descriptions an agent reads mid-run. Across three harnesses, six models, and 14 model/harness combinations, attack success ranged from 12.6% to 80.9%, while agents still completed their legitimate tasks 75.0-97.6% of the time. High utility and high attack success are not in tension. Both happened together, constantly.

The benchmark splits agent-harness safety into six phases: Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control, and Incident Recovery. In all three harnesses tested, Harness Configuration was the weakest phase. Attacks won not by breaking into the system but by quietly changing security-sensitive settings inside a workflow the agent was already authorized to run. No alarms, because nothing looked unauthorized.

The finding that should change how you build: some configurations detected the malicious instruction in over 90% of runs and executed it anyway. Detection and enforcement turned out to be two different systems. Right now a lot of agent tooling treats spotting the risk as the same thing as stopping it.

If you're running agents with persistent settings or permission files, treat those files as a separate trust boundary. Write-protect them, version them, diff them before anything reads them, not just another artifact the agent can touch mid-task.

A related benchmark in the same corpus found that repo layout itself changes prompt-injection success rates. Together they point the same way: the structure around the agent matters as much as the model running inside it. Watching for a follow-up benchmark that measures block rate directly instead of detection rate. That's the number that actually tells you if a fix worked.