Two teams found a popular agent cost-cutter raises costs

Quesma's Terminal-Bench 2.1 test found that RTK, a shell-output filter for coding agents, raises the cost of agent runs instead of cutting it.

Teams paying per token for agent runs are the ones this burns. RTK has almost 80,000 GitHub stars on a promise of 60-90% savings, trimming noisy command output before the model reads it.

Average cost per task rose 1% on Fable 5.0 and 17% on DeepSeek V4 with RTK running. Pass rates dropped 1 to 2 points on both models. On DeepSeek, 44 of 58 tasks cost more with RTK on than off.

The math explains why. Terminal output is about 7% of Fable's input tokens and 26% of DeepSeek's, so perfect compression of that slice can't cut a bill by 60%. When RTK trims something the agent needs, the agent runs another command to get it back, and each extra turn re-sends the whole context.

RTK's own rtk gain report counts bytes removed, not dollars saved or tasks solved. Quesma's numbers show why that gap costs money. A JetBrains A/B test in July found RTK 7.6% more expensive at low effort, a second team reaching the same result with a different setup.

On September 11, RTK released v0.49.0, adding a content-addressed SQLite store so an agent can recall output its filters removed. The release also adds per-filter counters that flag filters whose output gets pulled back too often. That targets the exact failure Quesma measured. Nobody has rerun the benchmark on it yet.

The pattern isn't unique to RTK. NVIDIA's SoL-Pi, an MIT-licensed extension for the Pi agent, claims 45-49% fewer tokens and about a third lower cost. It holds about 94% of Pi's EdgeBench score. Early testers on r/LocalLLaMA were split: one called results on DeepSWE tasks "not promising," another flagged cache misses on local setups.

The only test that means anything is dollars per solved task, run with the tool on and off across enough tasks to see the spread. A bytes-saved counter isn't that test.