Turning up test-time reasoning makes agents brute-force harder, not think better, when their environment changes underneath them

arXiv 2608.02358·medium signalXBlueskyLinkedInCopy link

ScrambleToolBench is an interactive terminal benchmark that strips semantic cues from tool schemas so agents must discover tool behavior purely through trial-and-error, then injects mapping drift, stochastic action failures, and temporal execution windows to test whether they revise their hypotheses. Frontier models discover the initial mapping fine but fail to adapt: under structural change they show belief inertia or fall back to exhaustive search rather than deductive strategies like cycle tracing, and increasing test-time reasoning amplifies the expensive brute-force search instead of enabling recovery. Persistent memory reduces compounding errors but does not restore structural inference — the practical read is that if your agent operates against an API or environment that mutates, more thinking budget is the wrong lever and you need an explicit re-discovery or invalidation step.