SecTDD Across 2,705 Trajectories: Showing Security Tests Upfront Adds 19.3 Points of Joint Success but Regresses Two of Nine Model-Benchmark Conditions

arXiv 2608.09740·medium signalXBlueskyLinkedInCopy link

This 2026-08-10 study treats security tests as executable specifications and separates three factors — whether tests are shown upfront, whether failures trigger revision, and how failures are represented — across 2,705 trajectories, 31 task instances, three secure-code benchmarks, 16 CWE categories, and two model families. Showing all visible tests upfront lifts hidden functional-and-security joint success by 19.3 percentage points on average but helps only seven of nine conditions and hurts two. Structured feedback repairs 80 failing candidates with zero joint regressions; raw feedback repairs 83 but causes three regressions. Candidates passing all visible tests still fail hidden behavior families under every regime.