Reddit
llama.cpp is sitting on 1,433 open PRs, and a curated list of ~50 of them is where CPU and hybrid inference speedups are stuck
A 135-upvote r/LocalLLaMA post catalogs the open CPU/RAM/disk/hybrid PRs and discussions: an RFC for a MoE expert cache that keeps hot CPU-resident experts in VRAM with hybrid hit/miss execution (#24528), AVX2 large-batch IQ prefill (#27402), tiled mul_mat for k-quants (#27851), AVX-512/VNNI paths for Q5_K/Q6_K (#27590), x86 VNNI Q2_0 claiming 3x (#26348), a pshard runtime for plan switching and streamed weights (#22692), and streaming MoE routed experts from disk (#25294). A commenter notes the repo has PRs dating to 2023; another contributor links an adaptive-MTP PR (#27210) that auto-scales depth from 3 up rather than making you pick.
Source
↳ Follow the thread