Pin your agent harness version: 35 sequential releases of one CLI moved quality with the model held constant

arXiv 2607.03691·medium signal

A longitudinal study (arXiv 2607.03691, submitted 2026-07-04, revised 2026-07-20) evaluated 35 sequential Qwen Code CLI releases against 50 stratified SWE-bench Verified tasks while holding the underlying LLM fixed, tracing quality fluctuations to individual pull requests and architectural components. Major open-source harnesses ship more than two releases per day and accumulate thousands of issues within months, so the practitioner reflex of blaming the model for a regression is frequently wrong. Actionable: pin your harness version in CI, keep a small stable task set you re-run on every harness bump, and diff harness releases before you diff models.