A reproducibility audit found only 55.6% of LLM vulnerability-research artifacts run, and embedded oracles fire on patched builds 20 times out of 30.

arXiv 2608.09567 pre-registered a screen of 104 papers from 2023–2026; only 59 (56.7%) have a publicly reachable artifact. Executing an 18-paper sample plus all 102 cases of an anchor benchmark, 58/102 anchor cases carry a script-internal CVE ID that diverges from the declared directory CVE, and only 10/18 artifacts complete their declared workflow at first run. The killer: artifact-embedded oracles show 60% sensitivity and 45% specificity, meaning 20 of 30 patched-counterfactual audits still produce the claimed signal on the patched build. A trigger on a vulnerable build is not evidence of CVE-specific reproduction.