SecureForge: Telling a Frontier Model to 'Write Secure Code' and Listing the CWEs Still Yields Vulnerabilities 23% of the Time
Flagged by The Batch #365, the SecureForge paper measures the benign case rather than adversarial red-teaming: across 250 ordinary coding prompts, frontier models produce statically verifiable weaknesses 23% of the time even when explicitly asked for secure production code and handed the CWE list they'll be graded on — and 12.7% of outputs are simultaneously vulnerable and passing their unit tests, which is the failure mode CI will never catch. The pipeline finds benign prompts that elicit vulnerabilities, amplifies them via Markovian/MCMC sampling into a diverse corpus, then uses GEPA genetic prompt optimization against a static analyzer to evolve a hardened system prompt, cutting CWE rate up to 48% with no loss in test passage and transferring zero-shot to real in-the-wild agent prompts (+13.5% joint secure-and-passing rate). It needs only API access and a static analyzer, with an open toolkit at github.com/sisl/SecureForge; note the paper itself is arXiv 2605.08382 from May 2026, newly amplified rather than newly published.
↳ Follow the thread