'Please Perform a Comprehensive Security Audit' — Benchmark Shows Generic LLM Prompts Miss Business-Logic Vulnerabilities

AISafe·medium signal

AISafe benchmarked Claude Code, Codex, Codex Security, and their own tool against the Gokapi file-hosting app. Generic Claude Code prompts returned 24 findings with 1 of direct impact and high false positives; AISafe returned 20 findings with 9 of direct impact and zero false positives. Only purpose-built tooling caught business-logic flaws: cross-user data leaks via upload status streams, API key privilege escalation through incomplete permission revocation, and cross-user file deletion via insufficient authorization. The conclusion: finding quantity is a vanity metric — generic AI security prompts reliably miss the vulnerabilities that actually matter.