oMLX 0.6.0 Ships Distributed LLM Serving Across Multiple Macs — Qwen3.6-27B Goes 16.1 → 28.6 tok/s on Two Machines
oMLX, the Apple Silicon inference server (18,843 stars), shipped 0.6.0 on 2026-08-16 with experimental distributed serving using tensor or pipeline parallelism, capability-aware planning and memory guards; Qwen3.6-27B reached 28.6 tok/s across two Macs versus 16.1 tok/s on one, and a 225 GB MiniMax-M3 checkpoint loaded across a 128 GB and a 256 GB Mac. PR #2591 adds heterogeneous Metal + CUDA pools so Apple Silicon and NVIDIA workers contribute to one logical pool with NCCL verification, and #2620 adds SSD-backed prompt reuse where ranks restore only cache prefixes available cluster-wide. Decode throughput during concurrent prefill improved 1.6x to 43x. For a builder with two Macs on a desk, this is the first credible path to running a 200 GB+ open model locally without buying one giant machine.
Source
↳ Follow the thread