AI Agents Brute-Force Instead of Adapting to Tool Changes

Frontier models discover a tool's initial mapping fine, then either freeze on stale beliefs or brute-force search once the API's schema drifts, per ScrambleToolBench. The benchmark strips semantic cues from tool schemas first, so models can't lean on function names or descriptions. Then it injects mapping drift, random call failures, and shifting execution windows.

That's the failure mode for anyone running agents against production APIs that get versioned, rate-limited, or quietly restructured. It's the exact kind of drift ScrambleToolBench is built to simulate. Models work out the scrambled mapping fine at the start, since there's nothing named to help them. The trouble starts once that mapping changes underneath them mid-run.

Instead of updating their model of the tool, models show what the paper calls belief inertia: they keep acting on the outdated mapping. Or they drop reasoning and start trying calls until one works. Turning up test-time reasoning doesn't fix either failure, per the paper, it just funds a more expensive version of the same search.