Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-26 · high
GPT-4o's high-severity flaws fell to 13.6% while low-severity ones rose to 43.5%, and prompts sometimes rewrote code without saying so.
Why now: The paper posted to arXiv in August 2026, testing whether prompting alone can substitute for retraining on secure code.
Researchers ran 424 security-sensitive Python tasks through GPT-4o and LLaMA 3.1-8B under five security-focused prompts, then scanned outputs with Bandit and CodeQL, per a paper testing whether prompting alone fixes insecure code.
Structured prompting fixed the obvious problem. GPT-4o's invalid or noncompliant outputs dropped to as few as 37, down from 338 of 424 tasks. It's a real fix for teams whose prompts produce code the model won't even format correctly.
The security picture underneath didn't clean up the same way. GPT-4o's high-severity findings fell from 20.8% of flagged issues to 13.6%, while low-severity findings rose to 43.5%, up from 32%. Overall weakness counts across the five prompt variants stayed roughly flat. Telling the model to be more careful didn't remove bugs so much as move them down the severity scale.
The paper's sharper finding is semantic drift, where stricter security prompts sometimes rewrote or deleted the exact unsafe construct a developer had asked for, without flagging the change. A prompt meant to harden the code changed what the code does, without saying so.
If a security prompt can silently edit your logic, code review has to check the diff against the original request, and not lean on the scan results alone. The paper doesn't say whether that drift ever introduces new bugs on top of removing the ones a developer wanted gone.
Each link below shares sources, entities, or timing with this story.
GPT competes with Claude / Shared entities / Same source domain / Shared topic / Earlier coverage
Linked by a graph relationship (GPT competes with Claude); both cover GPT, Python; reported by the same outlet (arxiv.org).
Claude Code benchmarked against GPT / Shared entities / Same source domain / Earlier coverage
Linked by a graph relationship (Claude Code benchmarked against GPT); both cover GPT, Python, Security; reported by the same outlet (arxiv.org).
Meta released Llama / Shared entities / Earlier coverage
Linked by a graph relationship (Meta released Llama); both cover GPT, Llama; earlier GPT coverage from 2026-05-02.
Linked by a graph relationship (Meta released Llama); both cover GPT, Llama; earlier GPT coverage from 2026-05-01.
Claude Code benchmarked against GPT / Shared entities / Earlier coverage
Linked by a graph relationship (Claude Code benchmarked against GPT); both cover GPT, Python; earlier GPT coverage from 2026-07-14.
Claude Code benchmarked against GPT / Shared entity: GPT / Same source domain / Earlier coverage / Tension
Linked by a graph relationship (Claude Code benchmarked against GPT); both cover GPT; reported by the same outlet (arxiv.org).
GPT competes with Claude / Shared entity: GPT / Same source domain / Shared topic / Earlier coverage
Linked by a graph relationship (GPT competes with Claude); both cover GPT; reported by the same outlet (arxiv.org).
OpenHands uses GPT / Shared entity: GPT / Same source domain / Earlier coverage
Linked by a graph relationship (OpenHands uses GPT); both cover GPT; reported by the same outlet (arxiv.org).