A single greedy decode manufactures self-improvement gains on a model that was never trained

arXiv 2608.20290·medium signalXBlueskyLinkedInCopy link

An audit of three rounds of LoRA self-training on Qwen3-8B against a frozen control model found seven systematic measurement failures, including a ledger built on one greedy decode showing capability changes on an untrained model, largely an artifact of inference batching. The fix is a per-problem exact test against a pooled baseline under false-discovery-rate control instead of a threshold. Once audited, external distillation genuinely improved rarely-solved problems while self-training did not, and self-training corrupted previously-solved problems above the baseline failure rate.