jundot/omlx Trending at 13.1K Stars: Apple Silicon LLM Inference Server With SSD KV-Cache Drops TTFT From 90s to 1-3s

GitHub·medium signal

oMLX is a local LLM inference server for Apple Silicon with a breakthrough two-tier KV cache system — hot cache in RAM, cold cache on SSD in safetensors format. When a previous context prefix recurs, cache blocks restore from disk instantly instead of recomputing, dropping time-to-first-token from 30-90s to 1-3s on long contexts. With a native macOS menu bar app, OpenAI-compatible API at localhost:8000, and optimized for coding agents (Claude Code, Cursor), it's gaining +187 stars/day at 13.1K total. Requires Apple Silicon M1+ with 16GB RAM minimum, 64GB+ recommended.