Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-05 · high
Requiring valid JSON alone conflicts with nine other instructions in the stack, and a free fix recovers up to 11 points for weaker models.
Why now: The paper landed in the Aug. 5 briefing, with the 96%-to-20% collapse as its headline result.
Instruction-following collapses from 96% to 20% once you stack 24 constraints on a single prompt, per a new arXiv paper testing three production-tier models.
That's enough missed instructions to break anything downstream that expects the model to do exactly what it's told, JSON output included.
The drop isn't random. The paper attributes it to specific pairwise conflicts between instructions, not general model fatigue. One rule causes the most damage: requiring valid JSON output conflicts with nine other instructions in the stack, more than anything else tested.
There's a fix, and it's cheap. A training-free instruction compiler resolves those conflicts before the prompt reaches the model, recovering up to 11 points of accuracy for weaker models. Strong models are left essentially unchanged, per the paper.
Worth remembering: the compiler is a patch for cheap models, not a general upgrade. If you're running cheap models for bulk work and asking for JSON alongside other instructions, you're probably losing some of those instructions to JSON itself. No amount of prompt rewriting fixes a conflict that's structural.
Each link below shares sources, entities, or timing with this story.
arXiv 2608.06370 evaluated models emitting code that calls tools against JSON-schema tool calling on BFCL v4. PTC matched or exceeded the baseline in 11 of 14 models, with the GPT-5.6 family up 10.6%, and held stable under parallel execution in 13 of 14. Under context degradat...
BFCL v4 results show PTC matching or beating JSON tool calling on 11 of 14 models, with the GPT-5.6 family up 10.6% and better stability under context degradation and parallel execution. Most agent frameworks hard-code structured output as the default. On current models that d...
Define permitted components via Zod schemas, LLMs generate constrained JSON that renders progressively during streaming. 36 pre-built shadcn/ui components out of the box. This is the cleanest production-tested path from "model output" to "rendered interface" available today.
ES achieves broader reasoning coverage, with verifier-projected Jensen-Shannon diversity across the ES population theoretically tied to higher Pass@K, and empirically improves Pass@1 while reaching higher Pass@K where GRPO collapses entropy. The proposed sequential GRPO-then-E...
arXiv 2608.23541 tested 11 verifier-scored optimization tasks under matched compute. Different model families do find structurally different solutions, and then a single round of reading each other's complete outputs erases exactly the diversity that justified using multiple m...
LivePlan watches a programming agent's trajectory with rule-based detectors that need zero model calls, waking an advisor LLM only on drift, repeated failed actions, or an imminent no-patch exit. On SWE-agent across five LLMs it raised resolution rates up to 15.2% (9.9% averag...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.