2,572 tests from 240 agent modules show the field tests agents wrong: narrow unit tests, heavy mocking, and simplistic inputs

arXiv 2608.08413·medium signalXBlueskyLinkedInCopy link

Tangent (arXiv 2608.08413, Aug 9) analyzed 2,572 test methods across 240 open-source LLM-agent modules and interviewed 10 industry practitioners, finding testing dominated by narrowly scoped unit tests with little integration or end-to-end coverage, mocking so heavy it obscures the real interactions that actually break, superficial validation, simplistic input patterns, and near-absent assessment of non-functional requirements. Commercial and open-source projects shared the same weaknesses, including no formal test framework and unclear test objectives. The output is a taxonomy of 23 testing patterns you can audit your own agent test suite against — useful right now if your agent tests mostly assert that a mocked subprocess was called with the right arguments.