Research
A Single Prompt on a Newer Model Beats the Engineered Tooling in 37-63% of ICSE 2026 LLM Papers
The authors take 35 LLM-based technique papers from ICSE 2026 and test whether each complex tool can be outperformed by the simplest possible alternative: one automatically generated prompt on a newer-generation model with no iterative refinement. For between 37% and 63% of papers it can. Constructive techniques such as code generation and repair are the most amenable to substitution, while a surviving set relies on strategies that feed the model additional insight and are amplified rather than obsoleted by newer models, which is a concrete test for whether a piece of scaffolding you are building is a workaround for a temporary model deficit.
↳ Follow the thread