Across 8,200 HumanEval Executions, JSON-Formatting Your Prompt Helped and LLM-Tuned Prompts Significantly Hurt Task Performance

arXiv (2608.21074, submitted 2026-08-21)·medium signalXBlueskyLinkedInCopy link

PromptResponse ran five semantically identical but syntactically distinct variants of HumanEval, baseline, JSON, Markdown, YAML, and an LLM-tuned version, through GPT-4o over more than 8,200 executions, measuring code performance, efficiency and stability. Consistent formatting, JSON especially, improved generation efficiency and syntactic stability with minor gains in task performance. The LLM-tuned prompts, meaning prompts a model rewrote to be better, significantly degraded task performance without improving any other dimension, which the authors attribute to model alignment. The dataset variants and evaluation pipeline are released.