A 250M Model Trained From Scratch Ships in 60MB, Runs 400 Tokens/Sec on Laptop CPU, and Pages 1-Bit Context to Disk

r/MachineLearning (112 upvotes)·medium signalXBlueskyLinkedInCopy link

A r/MachineLearning post details a 250M-parameter model trained on 30B fineweb tokens, quantized under 2 bits into a 60MB deployment needing roughly 80MB RAM, hitting about 400 tok/s on laptop CPU with no GPU. The last 2,048 tokens stay in fp16 KV cache while older context compresses to 1 bit on disk at about 320 bytes per token, so 1M tokens of history costs roughly 320MB, and the model was trained from the start to retrieve from that disk cache up to 100M tokens. The embedding table is 131k fixed 512-bit codes with zero trained parameters, 8.4MB total, scoring 0.619 Spearman on WordSim-353 against 0.029 for random codes; base quality is 3.15 nats/token, perplexity 23.3, 0.99 bits per byte.