Gwern's Technical Bet: Continuous Fine-Tuning, Not Context Windows, Is How a Personal Model Actually Learns You
The Guardian Angel essay stakes out a specific architecture against the prevailing RAG-and-long-context orthodoxy: dynamic evaluation (continuous fine-tuning on the user's own interactions) rather than frozen weights plus retrieval, paired with active learning where the model asks clarifying questions to accelerate personalization. He argues over-parameterized models with proper regularization resist catastrophic forgetting well enough to make this practical, and claims a 100x knowledge-worker productivity ceiling is theoretically reachable. The claim is unproven and the productivity number is the weakest part, but the architectural argument is a direct challenge to every 'memory' feature currently shipping as a vector store bolted onto a static model.
Source
↳ Follow the thread