Programmatic tool calling beats JSON tool calling on 11 of 14 models — the BFCL v4 head-to-head

arXiv·high signalXBlueskyLinkedInCopy link

Patel, Sen, Lumer and Subbiah (arXiv 2608.06370, submitted August 6) evaluated programmatic tool calling — the model emitting code that calls tools — against JSON-schema tool calling across 14 language models on BFCL v4. PTC matched or exceeded the JSON baseline in 11 of 14 models, with the GPT-5.6 family showing a 10.6% improvement, and held stable under parallel execution in 13 of 14 models; under context degradation the JSON baseline fell an average 2.3% while PTC held steadier. For builders the practical read is that the structured-output default many agent frameworks hard-code is no longer the safe choice on code-capable models, and the PTC advantage tracks model capability across release generations rather than fading.