JSON formatting beat model-rewritten prompts in GPT-4o test

PromptResponse ran GPT-4o through more than 8,200 code-generation executions, testing five HumanEval prompts that said the same thing in five different formats. The result complicates a common assumption in prompt engineering: that better wording, not formatting, is what makes an LLM produce reliable output.

Consistent formatting, JSON especially, improved generation efficiency and syntactic stability across the runs. Prompts that GPT-4o had rewritten to sound clearer went the other way: task performance dropped, and nothing else improved to offset it, according to the paper.

That points at a real risk in prompt-optimizer tools that hand your prompt to a model and ask it to improve the wording. In this test, the model was a poor judge of what instructions it should receive. Formatting moved the numbers, its own rewrites didn't.

The paper doesn't say why JSON specifically outperformed the other formats, or whether the effect holds outside HumanEval and GPT-4o. Five prompt variants on one model is a narrow base for a permanent house style, but it's cheap to test: pick a prompt you already use, wrap the same instructions in JSON, and compare output consistency before you let a model rewrite the wording for you. I'd run that comparison before trusting any prompt-optimizer's rewrite.