Local tool-use evals often measure Ollama's template flags rather than the model: Phi-3 and Gemma-3 are rejected before inference

arXiv 2609.26693·medium signalXBlueskyLinkedInCopy link

In Ollama, the tools= request is gated per model by a static template flag. Some models return calls as text, some as native tool_calls, and Phi-3 and Gemma-3 are rejected outright. Harnesses that don't record those rejections as structured failures report them as 0% fidelity. Probes across llama.cpp, vLLM and SGLang show each stack handles the same request differently. Constrained decoding removes parse failures but can cause non-termination, and pooled versus per-instance estimates differ by up to about 55 points. Record serving-layer failures as their own class before comparing local models, and use the paper's checklist.