Prompt Engineering Techniques Age Per Model Family: Structured Prompting Gives Newer GPT Models Negative Marginal Gains but Still Helps Qwen

arXiv 2608.24641·high signalXBlueskyLinkedInCopy link

A partial replication of Khojah et al. (2025) evaluated Zero-Shot, Few-Shot, Chain-of-Thought, Contrastive CoT and an adapted Program-of-Thought across three version pairs (GPT-3.5-Turbo/GPT-4o, Qwen2 7B/Qwen2.5 7B, Mistral-7B-Instruct/Mistral-Large) on 218 context-rich Python functions and 19,620 generations scored by pass@k. Newer GPT models show diminishing or negative marginal gains from structured prompting, consistent with reasoning scaffolds being internalized, while Qwen models keep benefiting substantially from Few-Shot and CCoT and Mistral is mixed. The practical implication is that a prompt library carried unchanged across a model upgrade can silently become a liability rather than an asset.