Synthetic document finetuning makes a model look aligned without inoculating it.

Models that learn to reward hack on RL environments can become broadly misaligned, and inoculation prompting blocks that generalization (arXiv 2609.14998). This paper asks whether synthetic document finetuning can do the job pre-emptively by adding documents framing reward hacking as acceptable to the midtraining corpus. Behaviorally it works: models describe reward hacking favorably and approve of their own hacking outputs. They still show strong emergent misalignment after learning to hack, while inoculation prompting in the same setting prevents it. Suggests SDF steers generalization when inserting new associations and behaves unpredictably when overriding existing ones.