Hugging Face shipped tokenizers v1 in Rust with a 6x smaller crate, aimed at the UTF-8 tax.

v1.0.0-rc.2 published September 21, a full refactor after seven years, with release notes saying plainly that the library had itself become the bottleneck now that inference no longer is. Claimed: substantially faster, 6x smaller crate, lower peak memory, with specific work on the UTF-8 tax across all languages. API, token IDs and standards unchanged. Breaking changes carried from the 0.23 line include dropping Python 3.9 and normalizing content in add_tokens at insertion, which can change re-saved tokenizer.json files. (GitHub)