ToolHazard auto-generates executable injection environments, and where the payload sits matters as much as what it says

arXiv 2608.11878·medium signalXBlueskyLinkedInCopy link

ToolHazard (arXiv:2608.11878, Aug 12) replaces hand-built prompt-injection test environments with an automated pipeline — Environment Simulator, Attacker Agent, User Simulator — that produces runnable environments with discoverable injection points rather than fixed, pre-marked attack locations. The headline result is that injection timing and placement within the environment state materially change attack success, meaning benchmarks with hardcoded payload positions systematically under- or over-state exposure. Training data synthesized by the framework improved robustness on both ToolHazard-Bench and AgentDojo without degrading legitimate task performance, so it doubles as a hardening data source, not only an evaluation.