Fetching from the wire…
Infra2026-08-09 · source-backed
PR #24448 adds Q2_0 to ggml for CPU (ARM NEON plus scalar fallback), completing the Q1_0/Q2_0/Q4_0/Q8_0 family, primarily to serve PrismML's Apache-2.0 Ternary Bonsai models. Format packs 2 bits per weight with one fp16 scale per 64 weights mapping {0,1,2,3} to {-1,0,+1,+2}·d. On an M4 Pro at 8 threads the 1.7B goes from 3.20 GiB/48.70 t/s at F16 to 461.79 MiB/117.20 t/s; the 8B from 15.25 GiB/14.88 t/s to 2.15 GiB/30.14 t/s. Mean KLD 0.00012–0.00020, 99.31–99.39% identical top-1 tokens. The asymmetry to check before swapping: token generation roughly doubles but prefill is slightly slower than F16 (170.51 vs 200.15 t/s at pp512). x86, Metal, CUDA, and Vulkan staged for later PRs.
Each link below shares sources, entities, or timing with this story.
Ternary Bonsai released PrismML / Shared entities / Shared topic / Earlier coverage / Tension
Linked by a graph relationship (Ternary Bonsai released PrismML); both cover Apache, FP16, PrismML; overlapping topics (fp16, model, token).
Ternary Bonsai released PrismML / Shared entities / Same source domain / Shared topic / Earlier coverage
Linked by a graph relationship (Ternary Bonsai released PrismML); both cover Apache, PrismML; reported by the same outlet (github.com).
Ternary Bonsai released PrismML / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Ternary Bonsai released PrismML); both cover Apache, PrismML; overlapping topics (bonsai, model).
Swift uses Metal / Shared entities / Shared topic / Earlier coverage
Linked by a graph relationship (Swift uses Metal); both cover CUDA, Metal, Vulkan; overlapping topics (cuda, model).
Swift uses Metal / Shared entities / Earlier coverage
Linked by a graph relationship (Swift uses Metal); both cover CPU, CUDA; earlier CPU coverage from 2026-06-26.
Swift uses Metal / Shared entities / Same source domain / Earlier coverage
Linked by a graph relationship (Swift uses Metal); both cover Apache, Metal; reported by the same outlet (github.com).
Shared entities / Same source domain / Tension
Both cover CPU, GiB, MiB; reported by the same outlet (github.com); pushes against this story (against).
Shared entities / Same source domain / Earlier coverage
Both cover CPU, CUDA, Metal; reported by the same outlet (github.com); earlier CPU coverage from 2026-08-02.