Cost-Performance Study: Which LLM Agent Design Choices Actually Help in Adversarial Environments

arXiv·medium signal

A controlled study of compound LLM agent design in CybORG CAGE-2 (a cyber defense POMDP) measures which design dimensions — context, reasoning method, and task decomposition — improve performance versus merely increasing inference cost. Practitioners lack guidance on these tradeoffs; this paper provides the first systematic cost-performance analysis across all three dimensions simultaneously. The finding that some expensive design choices add cost without improving outcomes is directly actionable for agent builders.