ML Repos' Own Validation Workflows Caught Only 2 of 23 Reproducibility-Breaking Mutations

arXiv 2608.27100·medium signalXBlueskyLinkedInCopy link

MLReproMutate applies controlled mutations to ML research repositories across four classes (random seed, dependency pin, data split, cross-validation fold count) and runs them against the validation workflows those repositories already ship. In an outcome-blind study of 39 frozen repository-operator cases, execution plus a bounded restoration procedure yielded 24 evaluable cases and 23 confirmed non-equivalent mutations, of which the existing workflows detected 2, an 8.7% detection proportion. The authors are careful that this does not make those repos irreproducible, only that CI passing tells you almost nothing about whether experimentally important choices are pinned.