The same model scores 0.00 or 0.96 on tool calling depending only on the serving adapter

This is the cleanest experimental result I've seen in weeks, and it explains a class of debugging pain I've hit personally.

Researchers held weights, test cases, decoding parameters and seeds fixed on BFCL v4 and changed exactly one thing: the serving adapter. The tool-call score moved across the entire available range, 0.00 to 0.96. Then they ran a 2x2 over chat template and parser and found both main effects at exactly zero, with all of the effect living in the interaction. arXiv 2609.03966

Sit with that for a second. Testing the template alone shows nothing. Testing the parser alone shows nothing. Only the specific pairing breaks, which is precisely why nobody catches this by evaluating components separately. It's the worst possible shape for a bug: invisible to every reasonable isolation strategy.

The downstream numbers get more concrete. On tau-bench's 115 retail tasks, the same adapter swap moves server-parsed tool calls from 0 to 636. Inside verl's AgentLoop at 7B, 45 of 115 generations contain a complete, well-formed tool call and exactly 0 are accepted or executed. The model did the work. The stack threw it away and reported a capability failure.

The authors ship a 98-line preflight check that catches every silent failure they found. Run it. If you're serving open weights behind vLLM, SGLang, or any adapter you didn't write, you have no idea whether your tool-calling number reflects the model or the plumbing, and that check costs you five minutes.

This finding does not stand alone today. ARC Prize ran GPT-6 Astra on ARC-AGI-3 Semi-Private under two harnesses. The Standard harness, which forces the model to keep visible notes about strategy and state, scored 62.7% for $26,098. A Provider Adapter harness that preserves opaque reasoning state between requests and compacts long conversations scored 99.9% for $18,817. On the 167 game-reasoning pairs both solved, the Provider Adapter runs were 3.66x faster and used 49% fewer total tokens. ARC Prize

Same model. 62.7 or 99.9. ARC Prize will now label both harness conditions separately on its leaderboard, which is the right call and should embarrass anyone still publishing single numbers.

Third data point, from a different direction. The TrueForge team ran 14 DevRev Enterprise-Bench tasks three times each with a blind judge. Claude Managed Agents with Opus 4.8 solved 11/14 at $11.80 and 10.0M tokens per run. TrueForge with the same model solved 11/14 at $8.60 and 3.7M tokens, averaging 19 tool calls per task against 32. Swapping in GLM-5.2 gave 11.7/14 at $3.00 per run. r/LocalLLaMA

Three independent measurements, three different methods, one conclusion. The stack around the model produces your numbers. Not the weights.

The transferable technique from ARC Prize: stop round-tripping agent state through visible text notes. Carry provider-native reasoning state forward across turns with compaction. Every time I've built a scratchpad-in-markdown loop it was because that's the pattern everyone publishes, and the measured cost of that choice is 37 points of accuracy and $7,000.