A new paper proposes LLMs that grow their own weights

Most large language models freeze their feed-forward weights once training ends. A paper posted to arXiv argues that's the wrong design and proposes generating those weights on the fly instead.

The idea, laid out in arXiv 2609.18842, swaps the fixed parameter bank for a compact hypernetwork that turns live session data into weight modulations, borrowing structure from mixture-of-experts models. A Bayesian belief over the generator's latent code updates as the session runs, so the effective weights get re-derived continuously instead of locking in after the first forward pass. Storage stays constant. The weights themselves don't.

The pitch over plain in-context learning: amortized compute, a freed-up context window, and knowledge that persists across turns instead of getting pushed out when the context fills. If it works, that's a different way to give a model long-running memory without just making the context window bigger.

Here's the gap. The abstract lays out an evaluation protocol but reports no benchmark numbers. There's no accuracy comparison against a standard transformer, no latency measurement for the per-turn weight generation step, and no accounting for how much compute the hypernetwork itself burns. "Amortized" is a claim about where the cost moves, not proof that it's smaller.

Anyone building memory or personalization on top of an LLM should treat this as a direction to watch rather than a technique to adopt. The mechanism (regenerating weights instead of stuffing more tokens into context) is the part worth tracking. Whether it beats a well-tuned retrieval system on cost or latency is exactly what the paper hasn't shown yet.