Agent Attack Success Tracks the Model, Not the Harness

ActBench ran 24,000 attack trajectories against 15 models and six open-source cowork agent harnesses, and the model mattered more than the harness, per the August 10 benchmark.

That gap matters for anyone assuming a well-built agent wrapper keeps them safe. Attack success held between 73.7% and 94.4% no matter which harness ran the task; swapping the model alone pushed it down to 10.1%.

ActBench pairs 600 test cases across 213 scenarios with a benign task and an adversarial twin. Each twin keeps the same instructions, config, and starting state but slips in a payload the agent can reach through its own tools. It spans 15 risk behaviors, six execution spaces, and 48 web-service APIs. It scores what the agent does at each step of a trajectory, before the final response ever lands.

No harness tested got anywhere close to zero attack success, which undercuts the idea that scaffolding and permission checks are the primary safety layer. A related benchmark on the same thread argues the fix is splitting safety logic into separate pieces, not hardening the harness as one block.

Teams picking a coordination framework for safety are optimizing the wrong layer. This data says the model underneath decides whether an injected payload executes, and a harness swap won't move that number much. ActBench's code went public on GitHub on August 10, so any team running cowork agents can rerun the same 600 cases before trusting the wrapper.