llama.cpp merges a tiled CPU matmul for k-quants that runs prefill 3-7x faster on x86

GitHub·high signalXBlueskyLinkedInCopy link

PR #27851, merged 26 Sep, unpacks k-quant weights into 256x256 int8 tiles and runs a 16x16 VNNI microkernel over them. The old vec_dot path unpacked the same quants many times over. On an AMD 9950X3D with 8 threads at 8192x8192, q3_K went from 0.73 to 5.16 TFLOPS and q4_K from 1.06 to 4.97, about 2x the existing repack path, with lower RMSE. The gain disappears for token generation: break-even is at 32 rows and pure GEMV runs at 80% of stock speed, so the path only turns on above 64 rows. That makes it a prompt-processing speedup for CPU-only local inference.