A Continual-Learning Byte Model Trained on an 8GB Laptop GPU Cuts Catastrophic Forgetting by 99.84%
mini-AGI (github.com/volotat/mini-AGI, created September 19, 155 stars, 139 points on Hacker News) trains from scratch on an RTX 3070 Laptop with 8GB VRAM using two dense prelude blocks plus one recurrent block applied up to 24 times, PonderNet per-character halting, top-8 mixture-of-experts routing, and disk paging that keeps only 32 experts resident so parameter count is bounded by disk rather than VRAM. At 318.1M characters read it reports held-out loss of 0.8336 ± 0.0331 nats/char (1.2026 bits/byte) from a 540.1M-parameter pool of 169 experts at about 778 characters/second. The headline result is a hyperparameter one: setting the trunk learning rate to 0.1x the experts' rate dropped forgetting from +2.23 to +0.0067 nats after half a million characters of single-subject training.
↳ Follow the thread