A small amount of finetuning data erased trained alignment

A small dose of finetuning data wiped out a model's midtraining-installed motivation, according to a paper posted to arXiv.

Researchers ran the test at up to 110 billion parameters and 1 billion midtraining tokens. That scale matters for anyone treating midtraining as a way to install durable values before finetuning starts.

The paper continued pretraining on large volumes of alignment-relevant documents. It then tested whether midtraining could tip a model's motivation when post-training data left it ambiguous between two options.

Midtraining worked. It tipped the balance. Then a small slice of finetuning data suggesting a competing motivation wiped the effect out entirely.

The paper's second finding cuts deeper. Rules were only reliably learned when a demonstration of the rule appeared in midtraining or post-training data. A rule stated but never demonstrated didn't generalize. That undercuts the idea that midtraining teaches behavior a model never sees performed.

If a handful of finetuning examples can undo midtraining, midtraining was never installing a value. It was setting a prior that finetuning overwrites on contact. Watch whether labs leaning on midtraining as an alignment layer test it against finetuning data built to disagree with it. This paper suggests a handful of contradicting examples is enough to undo it.