Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-21 · high
A new benchmark found the attacker pool it built barely resembles the attacks current safety evals actually test for.
Why now: The paper posted to arXiv in July 2026, as agent teams keep shipping on single-turn safety scores it says miss most of the real risk.
Single-turn jailbreak attempts against AI agents fail almost every time, 0 to 1%. Give the same attacker fifteen rounds to adapt, and the success rate jumps to 5.4-14%, per a new paper called Adaptive Adversaries (arXiv:2607.18063).
That's the gap between what most vendors test and what a patient attacker actually does. A safety score built on single-turn testing can undersell an agent's real failure rate by five to fourteen points.
Claude Opus 4.6 and GPT-5.4 tied at 5.4% aggregate failure. The average hides the real finding: Opus failed 60% of the time on one scenario where every competitor held at 7%.
The paper also found only 13 of 21 test scenarios could tell defenders apart at all. Eight scenarios didn't distinguish a good agent from a bad one.
The multi-LLM attacker pool behind these numbers barely overlaps with existing jailbreak benchmarks, cosine similarity between 0.02 and 0.14, close to unrelated. If your vendor's safety eval and this adaptive pool are testing different things, the score you were handed doesn't reflect it. It doesn't show what an adaptive attacker does to your agent in production.
Watch whether other labs start publishing adaptive, multi-round attack numbers instead of single-turn pass rates. If they don't, ask why.
Each link below shares sources, entities, or timing with this story.
LLM uses OpenAI / Shared entities / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (LLM uses OpenAI); both cover Claude Opus, GPT, LLM, Opus; overlapping topics (benchmark, opus).
Opus built by Anthropic / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Opus built by Anthropic); both cover Claude Opus, GPT, Opus; overlapping topics (agent, claude, opus).
Linked by a graph relationship (Opus built by Anthropic); both cover GPT, LLM, Opus; overlapping topics (agent, benchmark, current).
Opus built by Anthropic / Shared entities / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Opus built by Anthropic); both cover Claude Opus, GPT, Opus; overlapping topics (agent, benchmark).
LLM uses OpenAI / Shared entities / Same source domain / Shared topic / Earlier coverage
Linked by a graph relationship (LLM uses OpenAI); both cover Claude Opus, GPT; reported by the same outlet (arxiv.org).
Claude Code uses Opus / Shared entities / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Claude Code uses Opus); both cover GPT, Opus; overlapping topics (agent, benchmark, claude, opus).
LLM uses OpenAI / Shared entities / Earlier coverage / Tension
Linked by a graph relationship (LLM uses OpenAI); both cover Claude Opus, GPT, Opus; earlier Claude Opus coverage from 2026-04-25.
LLM uses OpenAI / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (LLM uses OpenAI); both cover Claude Opus, GPT; overlapping topics (agent, benchmark, claude, opus).