Stop putting the code under test in your test-generation prompt — swap in an LLM-written spec docstring instead

arXiv 2607.22883·medium signalXBlueskyLinkedInCopy link

arXiv 2607.22883 (July 24) documents a 'misguidance effect': when an LLM is shown buggy code and asked to write unit tests, it does two bad things at once — it generates more tests asserting the erroneous behavior as correct, and it generates fewer bug-exposing tests. Buggy code skews the model's preference toward tests that ratify the bug. The fix is a one-line prompt change: replace the code-under-test in the prompt with an LLM-generated specification docstring so the model targets intended behavior, which the authors show reduces misguided tests, increases effective ones, and also improves multi-round feedback-driven test pipelines.