Industrial Post-Mortem: LLM Unit-Test Generation Fails on Compilation, and Static Analysis Beats LLM Self-Repair

arXiv 2607.19682·medium signal

A deployment report on CATGen — built from repeated real-world failures inside proprietary industrial projects — finds that compilation robustness, not prompt quality, is the binding constraint for LLM-generated unit tests in codebases with complex frameworks and cross-file dependencies. The fixes that worked were unglamorous: make project-level dependencies explicit rather than letting the model infer context, construct the test class skeleton deterministically, and replace iterative LLM repair loops with lightweight static analysis. Evaluated on industrial focal methods and Defects4J, it improves compilation success and structural coverage while cutting both generation time and token spend.