Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-23 · high
A deployment report on the CATGen tool found compilation success, not prompt quality, was the real bottleneck in complex codebases.
Why now: It's dated July 23 on arXiv, and the paper doesn't name the company behind CATGen's proprietary codebase.
CATGen's deployment inside proprietary industrial projects found compilation success, not prompt quality, was the binding constraint on LLM test generation, per a new arXiv report.
That's a different diagnosis than most test-gen pitches make. In codebases with complex frameworks and cross-file dependencies, generated tests failed before they ever ran, and better prompting didn't fix it.
The changes that worked were mechanical, not clever. Project-level dependencies got made explicit instead of left for the model to infer from context. Test class skeletons got built deterministically instead of generated. The iterative LLM repair loop, the thing that keeps re-feeding the model its own compiler errors, got replaced with lightweight static analysis.
The report credits that swap with better compilation success, better structural coverage, shorter generation time, and fewer tokens spent per test.
I've hit this same wall building my own tools. A model doesn't need another retry loop when it fails to compile, it needs a harness that doesn't let it fail that way to begin with. Static analysis catching a broken import beats five rounds of "here's the error, try again."
The fix was a stricter harness, not a smarter model, deterministic scaffolding for dependencies and test skeletons, with static analysis standing in for repair loops. If that holds outside one company's proprietary repos, it argues against burning tokens on self-correction loops and for building the parts that don't need judgment as deterministic code instead.
The paper is dated July 23 on arXiv, and it doesn't name the company behind CATGen's proprietary codebase.
Each link below shares sources, entities, or timing with this story.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage / Tension
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-18.
LLM uses OpenAI / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (LLM uses OpenAI); both cover LLM; earlier LLM coverage from 2026-06-19.
Simon Willison released LLM / Shared entity: LLM / Earlier coverage
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-19.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-07-14.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-22.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-10.
Linked by a graph relationship (Simon Willison released LLM); both cover LLM; earlier LLM coverage from 2026-06-10.