Enabling web search cut benchmark accuracy by 8 points, and 21% of prompts gave inconsistent results across runs.

An audit of LLM benchmark methodology (arXiv 2608.06202, Aug 6) ran 401 stratified prompts from BBQ and SafetyBench through both ChatGPT's chat UI and the OpenAI API, with and without web search, collecting 4,812 responses over three repeated runs. Chat UI was less accurate than API on both benchmarks with search off. Enabling search cut accuracy by up to 8 points and reversed the direction of the modality trend on one benchmark. Repeated runs of the same prompt disagreed on up to 21%. Citation grounding and abstention diverged between modalities too. Every single-modality single-run accuracy number used to argue deployment readiness is measuring one narrow slice.