Anima Anandkumar on why transformers cannot touch industrial physics: context lengths in the hundreds of billions

Latent Space·medium signalXBlueskyLinkedInCopy link

In a Latent Space episode published 2026-08-26, Caltech's Anima Anandkumar argues the physical sciences resist the scaling recipe: open datasets in these domains hold tens or hundreds of thousands of examples, and at industrial resolution, a few hundred grid points per dimension, the context length runs to hundreds of billions or a trillion tokens, so all the world's compute would not be enough for a transformer. Her answer is Neural Operators, which learn a function evolving across scales rather than a fixed grid, with the spherical variant powering FourCastNet 3 by learning in the spherical harmonic basis and staying stable for months of rollout instead of days. In fusion, a few thousand samples predict plasma disruptions a million times faster than traditional simulation. The episode also covers TorchLean, which lets you write PyTorch-style networks inside the Lean proof assistant and formally verify them.