BF16 vs FP16 Greedy Decoding Diverges on 49-100% of Prompts, and Selective FP32 LM-Head Recompute Restores 22-36 Points of Agreement

arXiv·medium signalXBlueskyLinkedInCopy link

Across six 1.1B-7B models from four families, the same prompt under greedy decoding gives different outputs in BF16 and FP16 on identical hardware for 49-100% of prompts, and one token flip often cascades into a different trajectory. The flip is governed by the top-two logit margin at the LM head, not accumulated layer error. Broader FP32 compute makes agreement worse. Recomputing only the LM head in FP32 when the margin is small adds 22-36 pp exact agreement on A10G, which matters for anyone relying on 'temperature 0' reproducibility in evals or regression tests.