Distilled byte models match a token model on one-sixth the data and cut logit storage to a fifth.

This sweep trains ~1B layer-matched decoders across tokens, bytes, and bytes-with-end-of-token up to 1 trillion bytes, with two methods for converting token logits to byte logits. Token models win at low FLOPs then plateau; byte models start worse and pass them with more compute, with scaling-law extrapolation predicting distilled End-Of-Token-1B beating distilled Token-1B by up to 4% asymptotically. Operating over 256 bytes instead of a ~100K vocabulary also removes top-k truncation when dumping logits.