A case study watched one coding agent build a real data system against a fixed spec, and catalogued what it got wrong

arXiv·medium signalXBlueskyLinkedInCopy link

arXiv 2609.01985 (2026-09-02) fixes storage technologies, schema, entity-resolution algorithm and retrieval-filtering strategy in advance, then characterizes how an LLM coding agent behaves on systems-level requirements: schema design, async orchestration, configuration correctness, and retrieval-filtering tradeoffs. It is a defect taxonomy for the class of work agents are increasingly handed end-to-end, and it also argues about evaluation rigor, since the defects it finds are ones a passing test suite does not surface.