IDE-generated unit tests run but do not test: weak assertions outnumber outright failures
VibeCheck evaluated Kiro, Antigravity and Cursor (all with Claude Sonnet 4.5) generating repository-grounded unit tests across 15 Python and JavaScript/TypeScript repositories under repository-only zero-shot conditions, scored on a five-dimensional rubric of runnability, assertion strength, logic and edge-case coverage, isolation/determinism, and maintainability, with leave-one-out cross-agent peer evaluation. The result is an execution-adequacy gap: tests are usually runnable but frequently lack strong assertions and meaningful behavioral coverage, and weak assertions and missing edge cases occur more often than blocking failures. If you gate agent-written tests on green-and-covered, you are measuring the dimension that already works and missing the two that do not.
↳ Follow the thread