Attack Success Rate Is Not a Number: 65% of 259 Agent-Security Papers Report No Variance, and 100-Case Benchmarks Can't Detect Gaps Under 18 Points

arXiv·high signalXBlueskyLinkedInCopy link

A full-text meta-analysis of 259 agentic-security arXiv papers (Feb 2025 to Sep 2026) found 65.3% report no variance or repeated runs for their headline ASR. Only 30.9% disclose enough decoding detail to show whether the evaluation was stochastic, and 29.7% of LLM-judge papers check agreement with humans. On a 100-instance benchmark the minimum detectable ASR difference at conventional power is 18.2 points, so most published defense rankings with small gaps are noise.