Eleven days of production data: three quarters of small-model database agent failures happened after the model had already called a tool

arXiv 2609.21341·high signalXBlueskyLinkedInCopy link

The authors drove the agent mode of an open-source SQL client with 39 locally served open-weight models plus one hosted control across six task surfaces, logging 8,199 runs, 110,711 ledger events and 14,008 refused tool calls. Of 2,100 model-attributed losses, 1,590 (75.7%) came from runs that had invoked at least one tool, a split that holds in 99.7% of clustered resamples; transport failures (tools used, no deliverable) are the largest class at 36.2% and pure capability failures (no tool call at all) the smallest at 17.3%. Production ledgers recorded refusal codes but never the model's arguments, hiding the cause for ten days, and capturing arguments exposed five server defects including one tool that demanded a field its sibling forbade, so five server-side changes touching no model moved the numbers.