Carve regression tests from recorded production traffic plus a natural-language scenario — 85.4% adoption over a 9-month deployment

arXiv 2607.24000·medium signalXBlueskyLinkedInCopy link

arXiv 2607.24000 (July 27) is an experience paper on NL2Test, deployed in production for nine months starting March 2025, generating 3,196 microservice regression tests. The pipeline takes a natural-language scenario description plus captured traffic, uses the LLM only for semantic interpretation and constrained code synthesis, and hands the reliability-critical parts to deterministic algorithms: request filtering, dependency validation by value consistency, and assertion-path validation. Results were 85.4% code adoption, 82.4% exact match (42 of 51 scenarios), and 98.0% producing usable drafts with minor edits. The key design choice worth copying is binding dynamic values from prior responses rather than hard-coding them, and excluding non-deterministic fields from assertions.