ConvexTok: Tokenization via Convex Optimization Beats BPE and Unigram on Language Modeling

arXiv·medium signal

ConvexTok formulates tokenizer construction as a linear program solved with convex optimization, replacing the greedy local decisions of BPE and Unigram with globally optimal vocabulary selection. The new algorithm consistently improves intrinsic tokenization metrics and bits-per-byte (BpB) achieved by downstream language models. Practically significant because tokenization is one of the few unchanged components of the modern NLP pipeline — a principled improvement here compounds across all downstream tasks.