llama.cpp Merges Gemma 4 Tokenizer Fix: Newline Splitting Bug Was Silently Degrading Output Quality

r/LocalLLaMA / GitHub·medium signal

PR #21343 merged into llama.cpp main branch fixes a tokenizer bug where '\n\n' was incorrectly split into two separate tokens (107, 107) instead of the trained single token (108). The fix is C++-only with no GGUF regeneration needed. With 272 upvotes, the community flagged this as critical since the model was trained on multi-newline tokens and incorrect tokenization silently degraded longer conversations.