157 open-source agent projects with 100+ stars audited: safeguards are applied inconsistently across equivalent execution routes

arXiv 2609.17698·medium signalXBlueskyLinkedInCopy link

This study reads documentation, source, configuration and tests across 157 LLM-agent projects to characterize what QA actually exists, and finds coverage fragmented in a specific and checkable way. Guards are applied on one execution route but not on an equivalent alternative route to the same effect, tests rarely exercise boundary, adversarial or multi-step tool-use failures, and identified risks in a project's own docs are seldom turned into end-to-end checks. The actionable move for a builder is an audit that enumerates every route to a consequential action (direct tool call, code execution, shell, an MCP server exposing the same capability) and confirms the guard sits on all of them rather than on the documented one.