Tools
llama.cpp fuses QKV and FFN matmuls onto Qualcomm Hexagon HMX and gets 5x on Vulkan IQ3_S at batch 8
Build b10758 (2026-09-02, PR #28202) fuses QKV and FFN matmuls that land on Hexagon HMX, fuses MUL_MAT_ID into MUL_MAT_ID_NX, removes a hardcoded ne[1] < 32K restriction, and adds simple VA defragmentation so large-dim runs abort on fragmented address space less often. Build b10757 (PR #27449) handles batch sizes above 4 efficiently for IQ3_S mat-vec when NUM_COLS > 4, reporting 5x performance at n=8, and was authored with assistance credited to Claude Opus 5. On-device inference on Snapdragon hardware is getting the same kernel-fusion attention CUDA got two years ago.
Source
↳ Follow the thread