Static Analysis Beat LLM Self-Repair on Test Generation

CATGen's deployment inside proprietary industrial projects found compilation success, not prompt quality, was the binding constraint on LLM test generation, per a new arXiv report.

That's a different diagnosis than most test-gen pitches make. In codebases with complex frameworks and cross-file dependencies, generated tests failed before they ever ran, and better prompting didn't fix it.

The changes that worked were mechanical, not clever. Project-level dependencies got made explicit instead of left for the model to infer from context. Test class skeletons got built deterministically instead of generated. The iterative LLM repair loop, the thing that keeps re-feeding the model its own compiler errors, got replaced with lightweight static analysis.

The report credits that swap with better compilation success, better structural coverage, shorter generation time, and fewer tokens spent per test.

I've hit this same wall building my own tools. A model doesn't need another retry loop when it fails to compile, it needs a harness that doesn't let it fail that way to begin with. Static analysis catching a broken import beats five rounds of "here's the error, try again."

Less model, more scaffolding

The fix was a stricter harness, not a smarter model, deterministic scaffolding for dependencies and test skeletons, with static analysis standing in for repair loops. If that holds outside one company's proprietary repos, it argues against burning tokens on self-correction loops and for building the parts that don't need judgment as deterministic code instead.

The paper is dated July 23 on arXiv, and it doesn't name the company behind CATGen's proprietary codebase.