A 28.9M-Parameter LLM Runs on an $8 ESP32-S3 at 9.5 tok/s Using Per-Layer Embeddings to Keep 25M Params in Flash

GitHub (slvDev) via Hacker News, 245 points·medium signal

slvDev's esp32-ai (MIT, 1.2k stars) runs a TinyStories-trained 28.9M-parameter model on an ESP32-S3 with 512KB SRAM, 8MB PSRAM and 16MB flash, printing generated text to a small wired screen at about 9.5 tok/s end-to-end (9.7 tok/s pure compute). The trick is a Per-Layer Embeddings architecture that parks a 25M-parameter lookup table in flash while only the compute core lives in fast SRAM, with the whole thing quantized to 4 bits at 14.9MB. It's plain C built on Karpathy's llama2.c, and it cannot answer questions or follow instructions — it only writes short stories, which is exactly the honest framing that makes the memory-hierarchy trick worth studying.