XML system prompts cut answer quality from 74% to 48%

A production tender-response system's answer quality fell from 74% to 48%, per a new arXiv paper. The system prompt had swapped prose for nested XML tags.

That result cuts against the default most agent builders reach for, wrapping instructions in <rules> and <instructions> tags because vendor docs implied it helps. Structured markup still helped when the model was reading source documents. It only hurt once the instructions themselves got the same treatment.

The same paper found a second problem. Naming a forbidden construction in a prompt raises the odds the model produces it. Tell a model never to write a specific closing phrase, and that exact phrase turns up more often afterward.

The system still held up on the broader test. An LLM judge rated its answers as good as or better than the human-submitted version on 40 of 55 ground-truth sections. Of the 15 losses, only 6 traced to writing quality at all.

A 232-upvote r/ClaudeAI thread found the same pattern without citing any research. Correct Claude Code for adding ketchup to a coffee order. The fix gets written into the spec as 'make a coffee (without ketchup)' forever after. One reply in that thread disagrees, arguing the leftover negative constraint stops the model from re-adding what got removed. Both effects are probably real. The paper's numbers say the cost of the first outweighs the benefit of the second.