Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-04 · high
ScrambleToolBench found that extra test-time reasoning doesn't help models spot a tool's mapping change, it just buys a longer, costlier search.
Why now: ScrambleToolBench is a new benchmark, not a retest of an existing one, and it made the August 4 briefing as arXiv preprint 2608.02358.
Frontier models discover a tool's initial mapping fine, then either freeze on stale beliefs or brute-force search once the API's schema drifts, per ScrambleToolBench. The benchmark strips semantic cues from tool schemas first, so models can't lean on function names or descriptions. Then it injects mapping drift, random call failures, and shifting execution windows.
That's the failure mode for anyone running agents against production APIs that get versioned, rate-limited, or quietly restructured. It's the exact kind of drift ScrambleToolBench is built to simulate. Models work out the scrambled mapping fine at the start, since there's nothing named to help them. The trouble starts once that mapping changes underneath them mid-run.
Instead of updating their model of the tool, models show what the paper calls belief inertia: they keep acting on the outdated mapping. Or they drop reasoning and start trying calls until one works. Turning up test-time reasoning doesn't fix either failure, per the paper, it just funds a more expensive version of the same search.
Each link below shares sources, entities, or timing with this story.
OpenAI released Frontier / Shared entity: Frontier / Shared topic / Earlier coverage
Linked by a graph relationship (OpenAI released Frontier); both cover Frontier; overlapping topics (agent, budget).
OpenAI released Frontier / Shared entity: Frontier / Earlier coverage
Linked by a graph relationship (OpenAI released Frontier); both cover Frontier; earlier Frontier coverage from 2026-07-07.
OpenAI released Frontier / Same source domain / Shared topic / Tension
Linked by a graph relationship (OpenAI released Frontier); reported by the same outlet (arxiv.org); overlapping topics (against, agent).
OpenAI released Frontier / Shared entity: Frontier / Earlier coverage / Tension
Linked by a graph relationship (OpenAI released Frontier); both cover Frontier; earlier Frontier coverage from 2026-02-25.
OpenAI released Frontier / Same source domain / Shared topic
Linked by a graph relationship (OpenAI released Frontier); reported by the same outlet (arxiv.org); overlapping topics (better, reasoning).
OpenAI released Frontier / Shared entity: Frontier / Earlier coverage
Linked by a graph relationship (OpenAI released Frontier); both cover Frontier; earlier Frontier coverage from 2026-08-03.
Linked by a graph relationship (OpenAI released Frontier); both cover Frontier; earlier Frontier coverage from 2026-08-02.
Linked by a graph relationship (OpenAI released Frontier); both cover Frontier; earlier Frontier coverage from 2026-07-29.