Chaos-engineer your agent at the HTTP layer: AgentChaos injects LLM API faults with no code changes and drops pass@1 by up to 50 points

arXiv 2608.06790·high signalXBlueskyLinkedInCopy link

AgentChaos (ASE 2026) intercepts the HTTP interface between an agent and its model provider to inject crash, omission, and value faults into the content and tool_call fields of API responses — runtime fault injection with zero source modification, plus verification that each fault actually triggered. Across 65 fault configurations pass@1 fell by up to 50 percentage points, and rankings held constant across LLM backbones, meaning harness design mattered more than model choice. The sting: existing fault-diagnosis methods scored below 53% on fault type and below 56% on fault step, so most agent systems today cannot tell you *why* they broke.