Hugging Face shipped tokenizers v1 in Rust with a 6x smaller crate and a focus on the UTF-8 tax

GitHub / huggingface-tokenizers releases (announced on r/LocalLLaMA, 79 upvotes)·medium signalXBlueskyLinkedInCopy link

Hugging Face published tokenizers v1.0.0-rc.2 on 2026-09-21, a full refactor after seven years, announced on r/LocalLLaMA by HF's Aritra. The release notes say the library had itself become the bottleneck now that model inference no longer is, and claim it is substantially faster with a 6x smaller crate size and reduced peak memory, with specific work on cutting the UTF-8 tax across all languages. API, token IDs and standards are unchanged. Breaking changes carried over from the 0.23 line include dropping Python 3.9 and normalizing content in `add_tokens` at insertion, which can change re-saved `tokenizer.json` files.