One in seven scanner-clean Python samples still exploitable

Researchers ran 1,355 Python code samples through Bandit and Semgrep, then layered on LLM-based reasoning about CWE categories. Then they tried to exploit what was left, inside Docker containers, according to a paper posted to arXiv.

Of the 654 samples that came back clean from both scanners, 95 still had a working exploit. That's 14.53%, or roughly one in seven, for anyone treating a green SAST check as a merge gate.

Weak random number generation (CWE-338) and weak password hashing (CWE-916) showed up repeatedly among the confirmed exploits. Neither Bandit nor Semgrep flagged either category. Both are bugs that look fine at the syntax level: the code calls a hashing function, it just calls the wrong one, or calls random where it needed secrets.

Static scanners pattern-match against known-bad function calls and syntax shapes. They don't reason about whether a specific RNG or hash choice is sound for the context it's used in. An LLM given the same code and asked to reason about CWE categories caught more of it. The researchers then proved the gap wasn't theoretical by exploiting the samples directly.

The paper doesn't say whether adding an LLM reasoning pass before merge would have caught all 95 exploits or just some of them. What it does establish is where the blind spot sits: crypto and randomness choices, not the syntax patterns Bandit and Semgrep were built to catch.