Dispatch
Transformers can now load a llama.cpp GGUF file directly and hit near-llama.cpp throughput on Apple silicon
Hugging Face added packed GGUF inference to transformers: AutoModelForCausalLM.from_pretrained("unsloth/Qwen3.5-4B-GGUF", gguf_file="Qwen3.5-4B-Q4_K_M.gguf") and everything after that is the normal transformers API, with BF16, Q6_K, Q5_K_M and Q4_K_M supported. On a MacBook Pro M2 Max throughput came close to llama.cpp across three checkpoints, though the transformers figure includes prefill while llama-bench reports decode only. Limits worth knowing before you plan around it: Apple silicon and Metal only, padding and batching still rough, and architecture support starts at Qwen3.5 plus compatible Qwen3.8 models.
↳ Follow the thread