When Specs Conflict, LLMs Consistently Prefer Formal Over Natural Language Over Examples

arXiv·medium signalXBlueskyLinkedInCopy link

A controlled symmetry-based framework constructs specifications with explicit built-in conflicts so model choices between competing sources can be directly observed and attributed, using 550 conflict instances across 11 function families and four representation types. Results show systematic rather than random resolution, with a consistent ordering of Formal ≈ Naturalized Formal > Pure Natural Language > Input-Output Examples, and example effects that depend on model capability and function family. The framework extends to heterogeneous conflicts in Boolean algebra, code generation, and the clinical domain — practical guidance for anyone stacking system prompts, tool schemas, and few-shot examples that may disagree.