1,116 web apps, one controlled variable: a boot probe kills nearly every launch failure at a third of a shell's cost

More checkers do not make better software. Somebody finally measured it.

The study built a minimal coding agent and held everything constant except its verification surface, meaning the set of self-checking tools it could reach: linter, boot probe, shell, screenshot. Six models, eight tool configurations, 1,116 web applications, graded by a condition-blind human against a frozen rubric with automatic probes stress-testing API-observable behavior (arXiv 2608.28795).

The numbers are unusually actionable. Without verification tools, about 1 in 7 applications failed to launch at all. Adding a boot probe eliminated nearly all launch failures at about 35% of the token cost of a full shell. The full shell ran 2.35x the no-tools baseline. And screenshots contributed nothing to non-visual failures, the example being scroll performance on a 100,000-row list, which is exactly the kind of failure a screenshot cannot see and a human reviewer immediately can.

I have been granting agents the full toolbox by reflex. Shell, browser, screenshots, the lot. This says I have been paying 2.35x for capability that mostly addresses failures I do not have. The cheapest single check, does the thing start, removes the most common catastrophic outcome for a third of the price of the expensive one.

The rule I am taking from this: budget each verification tool against a failure mode you have actually observed in your own runs. Not the failure modes you imagine. Go read your last twenty failed agent sessions and count what actually broke. If most of them are "the app doesn't start," give the agent a boot probe and nothing else. If most of them are layout, the screenshot earns its slot. If most of them are logic errors that pass tests, no verification tool in this study helps you and you need a different intervention.

This sits next to two other pieces from the same window. LoopArena benchmarks the controller model separately from the coding agent it drives, so you can tell whether the loop or the worker failed. Infobip published the four-phase workflow its AI research team uses, front-loading human effort into research and planning on the observation that upstream errors compound while correcting generated code introduces bloat (arXiv 2608.30701). Three groups measuring loop decisions instead of asserting that the loop matters. The vocabulary for this work exists now, which it did not six months ago.

Infobip is refreshingly honest about the gap: there are no metrics for workflow effectiveness, and the formalized context-management components in the literature do not map to what practitioners need at the workflow level. Both true. Both a problem.