Truncating old tool results beat a model upgrade by 21 points

Same weights. Same tasks. Same time budget. One change to the harness, and fail-to-pass on SWE-bench Verified went from 28% to 49%.

The setup in arXiv 2608.26218 is clean enough to be worth describing precisely. 169 SWE-bench Verified tasks, a 20,480-token context window, a fixed 480-second cutoff. The control keeps the full time-ordered conversation. The treatment shortens older tool results as the context fills, and reacts when the agent repeats itself or stalls. Mean per-task fail-to-pass fraction moved from 28% to 49%, and complete solutions went from 43 to 72 out of 169. The same frozen treatment transferred to three other model designs with no retuning.

Twenty-one points of fail-to-pass is a generational gap. Terminal-Bench-Science, released this week, shows Claude Opus 5 at 30.0% against Opus 4.8 at 10.5% (Terminal-Bench-Science). A harness change bought comparable movement for the cost of a truncation policy.

Three other papers this week point the same direction from different angles, which is why I'm giving this the slot rather than treating it as one result. PILOT (arXiv 2608.26530) adds a supervisor that can kill or redirect a running worker mid-execution and distills failures into reusable skills as they occur, gaining up to 9.8 points on Terminal-Bench 2.0 while cutting mean output tokens 42.9 to 47.4%. SKILL.state (arXiv 2608.26263) replaces the append-only conversation with a mutable structured state object, showing the model only the skill spec, the current state and the latest observation, and discarding intermediate reasoning once it's produced a validated state update. A manager-worker study across nine models from 9B to about 2.8T (arXiv 2608.26480) found the scaffold buys accuracy more cheaply than moving to a larger model, though it's null or negative for a third of the models tested.

All four are attacking the same thing: the conversation is a bad data structure for long-horizon work, and every token of stale tool output you carry forward costs you both money and attention.

Implement the cheap version this week. In your agent loop, when context crosses some threshold, replace tool results older than N steps with a one-line summary of what the call was and whether it succeeded. Keep the most recent ones intact. That's the whole intervention in the paper, and it's maybe thirty lines in a typical harness. The manager-worker result is worth knowing before you reach for a bigger model, and the caveat matters: it doesn't help every model, so measure before you commit to it.

There's a second claim in that paper I want to sit with. The authors argue that coding-agent evaluations must name the model and the harness as the tested solver, because reporting "Opus 5 scores X on SWE-bench" without the harness is reporting an underdetermined number. Given a 21-point swing from truncation policy alone, they're right, and roughly every benchmark table I've read this year is missing half its independent variable.