AFL++ fuzzed agent-written reimplementations of ten Linux utilities: fewer memory errors than the shipped versions, but more infinite loops
Rather than scoring an agent on a benchmark, this team pointed a best-practices agentic workflow at ten release-quality Linux utility programs, regenerated them, and measured reliability with both classic generational black-box fuzzing and AFL++ coverage-guided mutational fuzzing against the real distribution versions as ground truth. The AI-generated versions were typically as reliable or more reliable than the latest human versions, with a specific failure-shape shift: fewer memory errors such as buffer overflows, more hangs such as infinite loops. The authors also report the result is highly sensitive to the prompts and skills used and to how the supervising human responds, and argue the workflow's prompts and skills become a usable specification of the code. The practical read is to add hang and timeout detection to your agent-output test harness, since the class of bug shifts rather than disappearing.
↳ Follow the thread