Context compression looks free on completion metrics and isn't: retrieval calls nearly tripled (21.0 to 63.9) at 5x compression

arXiv 2608.16370·medium signalXBlueskyLinkedInCopy link

Across three models and two environments over a 24-turn horizon, 5x context compression produced no statistically significant change in task completion — but all six model/regime comparisons showed more retrieval calls, five significant after correction. For GPT-5.5, completion nominally rose 80% to 85% (p=1.0) while retrieval calls went from 21.0 to 63.9 (p=.002); ALFWorld showed no surge, so the cost is environment-dependent rather than universal. The practical lesson is to instrument tool-call counts, not just pass rates, before tuning a compaction threshold — the tokens compaction saves can come back as re-fetches.