Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-08-05 · high
Requiring valid JSON alone conflicts with nine other instructions in the stack, and a free fix recovers up to 11 points for weaker models.
Why now: The paper landed in the Aug. 5 briefing, with the 96%-to-20% collapse as its headline result.
Instruction-following collapses from 96% to 20% once you stack 24 constraints on a single prompt, per a new arXiv paper testing three production-tier models.
That's enough missed instructions to break anything downstream that expects the model to do exactly what it's told, JSON output included.
The drop isn't random. The paper attributes it to specific pairwise conflicts between instructions, not general model fatigue. One rule causes the most damage: requiring valid JSON output conflicts with nine other instructions in the stack, more than anything else tested.
There's a fix, and it's cheap. A training-free instruction compiler resolves those conflicts before the prompt reaches the model, recovering up to 11 points of accuracy for weaker models. Strong models are left essentially unchanged, per the paper.
Worth remembering: the compiler is a patch for cheap models, not a general upgrade. If you're running cheap models for bulk work and asking for JSON alongside other instructions, you're probably losing some of those instructions to JSON itself. No amount of prompt rewriting fixes a conflict that's structural.
Each link below shares sources, entities, or timing with this story.
Shared entities / Shared topic / Earlier coverage
Both cover JSON, LLMs; overlapping topics (json, model); earlier JSON coverage from 2026-03-29.
Shared entity: JSON / Same source domain / Shared topic / Earlier coverage
Both cover JSON; reported by the same outlet (arxiv.org); overlapping topics (constraint, model).
Shared entity: LLMs / Same source domain / Earlier coverage / Tension
Both cover LLMs; reported by the same outlet (arxiv.org); earlier LLMs coverage from 2026-07-31.
Both cover LLMs; reported by the same outlet (arxiv.org); earlier LLMs coverage from 2026-07-28.
Both cover LLMs; reported by the same outlet (arxiv.org); earlier LLMs coverage from 2026-07-26.
Shared entity: LLMs / Same source domain / Earlier coverage / Downstream implication
Both cover LLMs; reported by the same outlet (arxiv.org); earlier LLMs coverage from 2026-06-19.
Both cover LLMs; reported by the same outlet (arxiv.org); earlier LLMs coverage from 2026-06-19.
Both cover LLMs; reported by the same outlet (arxiv.org); earlier LLMs coverage from 2026-06-14.