Tools
Q1_0 ARM repack kernels give a 1.7B model 3.5x prompt processing on a phone
llama.cpp PR #23492 (merged 2026-09-21) adds 4x4 and 4x8 ARM NEON repack kernels for Q1_0, targeted at Bonsai LLM models. On a Snapdragon 7 Gen 3 (P cores only) with Bonsai-1.7B, the NEON+DP 4x4 path moves pp128 from 27.17 to 102.12 t/s (+276%) and tg32 from 20.80 to 35.96 t/s (+73%); the NEON+DP+I8MM 4x8 path reaches 121.87 t/s pp128 (+349%) but is slightly slower at token generation. The author published KL-divergence and perplexity tables alongside the speed numbers — mean PPL(Q)/PPL(base) of 1.0055 and 99.2% same-top-p — which is the quality evidence most quantization PRs skip.
Source
↳ Follow the thread