FreeToken Serves 290B+ Parameter MoE Models on RTX 30/40/50 Consumer Cards Using CPU-GPU Co-Execution

GitHub / Hacker News·medium signalXBlueskyLinkedInCopy link

FreeToken is an Apache 2.0 edge-native MoE serving engine (922 stars, 60 forks, 30 recent commits on main) that targets gaming desktops and laptops rather than datacenter GPUs. Its techniques are bandwidth-adaptive CPU-GPU co-execution under an optimal policy, full-layer double-buffered prefill streaming, a global LRU expert cache, semantic-aware caching with checkpoints for recurrent state and KV, and dynamic VRAM reallocation between expert cache and KV memory. The README claims interactive speeds without publishing tokens/second, so treat the throughput claim as unverified; the accompanying paper is arXiv 2608.16157.