Store text as the model's own token IDs instead of UTF-8: 3.30x compression that beats every byte codec, including a corpus-trained zstd dictionary

arXiv 2608.02376·low signalXBlueskyLinkedInCopy link

This paper argues that agent-adjacent storage should keep text as BPE token IDs rather than UTF-8, eliminating repeated tokenize/detokenize translation at every read. Packing r50k IDs as uint16 gives 2.25x on English with no compression at all, rising to 3.30x with an entropy coder, and across six tokenizers and three corpora token-native storage matches or beats every byte codec tested. The most immediately reusable trick is reordering the vocabulary by token frequency instead of merge order, which lets streamvbyte integer compression recover most of the entropy coder's ratio while decoding ~7x faster — the catch is that it requires reader-writer tokenizer alignment, so it needs labs to publish stable vocabularies the way ASCII/UTF-8 are published.