Terminal-Bench 4.0 Shipped With a Flat 8-Hour Agent Timeout and 8 Tasks Removed
Terminal-Bench (tbench.ai), via r/LocalLLaMA·high signal
The Terminal-Bench team released 4.0, calibrating per-task time, CPU and memory into a single flat 8-hour agent timeout, removing 8 tasks (2 saturated, 2 refusals, 2 with public solutions, 2 quality or platform issues) and fixing 19 more for flakiness or misspecification. They call it a major version because the changes require re-running every trial. The r/LocalLLaMA thread (256 upvotes) read the new board as GLM-5.3 sitting level with Fable 5 inside margin of error, and the top skeptical comment argued the generation-over-generation jump is 'not just a saturation issue' given known benchmark leakage risk.