Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-09-20 · high
Both models passed all 20 test assertions, but the smaller one took three times longer to finish.
Why now: As of September 20, the write-up is the first to weigh a ternary quant's token cost against its VRAM savings.
A benchmark writeup pitted Qwen3.8 27B IQ3_XXS against Ternary-Bonsai-2-27B-PQ2_0 on the same four UI generation tasks, on one 16 GB card. Both models finished all four tasks and passed all 20 assertions. The smaller model still took three times longer and burned three times the tokens to get there.
Qwen3.8 runs at 10.18 GiB and finished its agent run in 8 minutes. Bonsai runs at 6.42 GiB, a difference of 3.76 GiB, but needed 24 minutes and 9 seconds to reach the same result.
The token counts told the same story. Qwen3.8 generated 27,197 output tokens for the full run. Bonsai generated 84,176, a 3.1x jump for the same 20/20 outcome.
Qwen3.8 also decoded faster on average, 83.59 tokens a second against Bonsai's 64.91. Its speculative decoding acceptance rate was higher too, 65.22% using MTP against Bonsai's 39.76% on a modified n-gram approach.
The test covered one card, one model pairing, and four tasks. It doesn't say whether the pattern holds for other ternary quants or longer agent runs.
Each link below shares sources, entities, or timing with this story.
Can a model that fits on a Raspberry Pi do reliable tool calling? Two independent labs just answered yes. PrismML emerged from stealth March 31 with Bonsai, the first commercially viable 1-bit LLMs built on Caltech research. The 8B model fits in 1.15GB (vs 16GB for FP16), runs...
Ternary lands at 5.9GB, 1-bit at 3.9GB, both Apache 2.0, with claimed preserved multimodal and agentic behavior. (Latent Space) A genuinely agent-capable model at consumer-hardware footprint under a permissive license is a real shift in what runs off-cloud, and it pairs with t...
On July 14, llama.cpp merged native support for Tencent's Hunyuan Hy3 architecture (PR #25395), a 295B-parameter, 21B-active MoE. Any recent master build can load it now. Community GGUF quants (Q2_K, IQ2_M, Q4_K_M) from AngelSlim and others already ship on Hugging Face, and so...
A developer published v100-skinny with hand-written NVFP4 W4A16 CUDA kernels plus chain-MTP speculative serving: four V100s at 219.1 ± 5.9 tok/s decode against a 5090 running NInfer at 214.7 ± 9.2, both 5/5 correct on AIME 2026 problem 1 across five seeds. (r/LocalLLaMA) The m...
Prism ML released it September 16 as GGUF and MLX builds storing embeddings, attention projections, MLP projections and the LM head as ternary {-1,0,+1} weights with FP16 group scaling, cutting a ~54 GB FP16 model to 5.95 GB while keeping 262K context. The card claims 98.2% of...
GSQ-RCO GGUFs for Qwen3.8-Flash-Next cut the model from roughly 80-95GB to 68-76GB, with IQ3_XXS matching the base model exactly on AIME25 and within 0.51 on GPQA-Diamond. The unusual variant is Q2_0, which trades 0.09 points of task average for 3.4x prompt throughput and 1.9x...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.