Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-25 · high
CWEAgent's benchmark accuracy was 85%, but it matched just 49.70% of labels across 15,556 real CVEs.
Why now: The audit's CVE data runs through 2026, its most recent year of disclosures.
CWEAgent audited 15,556 open-source CVEs disclosed from 2017 through 2026, matching just 49.70% of NVD's CWE labels to code-grounded ones, per a CWEAgent audit posted to arXiv.
Every scanner evaluation and machine learning tool that treats NVD's CWE field as ground truth inherits that gap. Reliability varies sharply by which organization assigned the CVE and what kind of weakness is involved.
On a curated 100-CVE benchmark, CWEAgent reached 85% top-1 accuracy. It uses a structured representation built to capture root cause, trigger condition, violated property, exploit mechanism, and impact. Across the full 15,556-CVE set, agreement with NVD's labels was just 49.70%.
At the larger 15,556-CVE scale, 31.37% of the mismatches are defensible under CWE's own taxonomy ambiguity. More than one label can reasonably apply to the same flaw. The remaining 3.63% look like outright errors, not disagreements about interpretation.
That gap concentrates in weakness types where the taxonomy itself is still contested.
Each link below shares sources, entities, or timing with this story.
Shared entities / Same source domain / Shared topic / Earlier coverage
Both cover CVE, CVEs; reported by the same outlet (arxiv.org); overlapping topics (cause, cves).
Shared entities / Earlier coverage
Both cover CVE, CVEs, Only; earlier CVE coverage from 2026-07-31.
Shared entities / Same source domain / Earlier coverage
Both cover CVE, CVEs; reported by the same outlet (arxiv.org); earlier CVE coverage from 2026-05-08.
Shared entities / Shared topic / Earlier coverage
Both cover CVE, CVEs; overlapping topics (cves, disclosed); earlier CVE coverage from 2026-04-21.
Shared entities / Earlier coverage / Tension
Both cover CVE, CVEs; earlier CVE coverage from 2026-05-10; pushes against this story (against).
Both cover CVE, CVEs; earlier CVE coverage from 2026-03-23; pushes against this story (against).
Shared entity: CVE / Same source domain / Earlier coverage / Tension
Both cover CVE; reported by the same outlet (arxiv.org); earlier CVE coverage from 2026-08-21.
Shared entity: Only / Same source domain / Earlier coverage / Tension
Both cover Only; reported by the same outlet (arxiv.org); earlier Only coverage from 2026-08-20.