Open Coding Models Are Overfit to OpenHands: Fine-Tuned Agents Degrade Under Any Scaffold They Weren't Trained On

arXiv 2608.06113·high signalXBlueskyLinkedInCopy link

DCAS (arXiv 2608.06113, Aug 6) documents that the open agent ecosystem collects nearly all fine-tuning trajectory data under a single scaffold, OpenHands — and models fine-tuned on it score well there but degrade substantially under any other CLI scaffold, while untrained base models show no such divergence. The authors pin the load-bearing variable on planning structure, separating explicit planning (a pre-execution plan artifact) from implicit planning (structural conventions throughout the agent loop). Their fix is a backend-substitution interception layer that routes API traffic between any CLI scaffold and any backend model without modifying the scaffold; a model fine-tuned on a small set of planning-aware trajectories from one scaffold then gains consistently across non-training scaffolds. If you benchmark open coding models, this says your numbers are partly measuring scaffold match.