Qwen Tokenizes 330 Lines of HTML/JS Into 1,609 Tokens; Gemma Needs 4,258 for the Same File — a 2.6x Gap That Vanishes on Prose

r/LocalLLaMA (215 upvotes / 48 comments)·medium signalXBlueskyLinkedInCopy link

An r/LocalLLaMA post (215 upvotes / 48 comments) ran identical inputs through Qwen 35B-A3B and Gemma 26B-A4B and found the tokenizers diverge almost entirely on code: 1,609 vs 4,258 tokens for a 330-line HTML/JS file, but 1,025 vs 1,039 tokens for a 55-line instruction document. That is a near-3x difference in effective context and cost on exactly the workload where Qwen is reputed to beat Gemma, and it suggests the coding-vs-language reputation gap between the two families is partly a tokenizer artifact rather than a pure capability difference. For anyone sizing context windows or per-token budgets on local models, this is a cheap measurement to replicate before choosing a base model.