Most Code-Generation Failures a Hint Rescues Were Already Reachable by Plain Resampling

arXiv 2609.01106·medium signalXBlueskyLinkedInCopy link

Testing on HumanEval+ and MBPP+ with executable evaluation, adaptive relevant hints rescue 36 of 79 selected Qwen2.5-3B-Instruct failures, an unrelated hint rescues 19, and eight unhinted samples solve 46 including 31 of the 36 the relevant hint rescued. Phi-3.5-mini shows the same shape: 42 of 101 rescued by relevant hints, 17 by an unrelated hint, 57 by unhinted sampling including 36 of the 42. Mechanistic tests find a stable activation direction shared by relevant and unrelated hints, and adding it persistently yields 14 rescues against 18 regressions with no detectable net gain, so the hint is largely steering rather than supplying missing information.