Nested constraints are where instruction-following breaks: the best of seven models barely clears 50% prompt-level accuracy, and it drops as depth grows

arXiv 2607.27912·medium signalXBlueskyLinkedInCopy link

IFHierBench targets the now-common pattern of one LLM call producing a layered artifact where the whole output, each structural section, and nested fields all carry their own constraints — something flat instruction-following benchmarks cannot score because they apply every check uniformly to the whole response. It ships 600 prompts stratified across four constraint-tree depths and 35 distinct constraints, each paired with a deterministic checker that verifies satisfaction at every scope. Across seven leading proprietary and open-weight models the strongest only marginally exceeds 50% prompt-level accuracy, with sharp degradation as depth increases — practical read: flatten your constraint tree or split deep nested requirements across calls rather than trusting one prompt to honor all levels.