oMLX: Apple Silicon-Optimized LLM Inference Server With Tiered KV Cache Reaches 10.4K Stars

GitHub·medium signal

oMLX is a purpose-built LLM inference server for Apple Silicon Macs featuring continuous batching and a tiered KV cache that splits between fast RAM (hot tier) and SSD storage (cold tier) for extended context persistence. At 10,366 stars with +234 today, it runs LLMs, vision-language models, embeddings, and rerankers simultaneously with automatic memory management — all controlled from a native macOS menu bar app (PyObjC, not Electron). Requires macOS 15.0+ and M1/M2/M3/M4.