Transformers adds a 780B MoE model with 1M-token context

Hugging Face's Transformers library picked up Hy4-Preview in its 5.17.0 release notes, a mixture-of-experts model with 780B total parameters and a 1M-token context window. Only 49B of those parameters activate per token, routed across 256 experts with top-8 selection plus one expert that's always on.

That gap between total and active size is what matters here. A 780B model that only wakes 49B parameters per token costs far less to run than its full size implies. That ratio is what makes a parameter count this large workable outside a handful of labs with unlimited GPU budgets.

What stands out is how many attention mechanisms land in one architecture at once. Hy4-Preview combines Multi-head Latent Attention with DeepSeek Sparse Attention and shared indexer layers. It adds gated MLA with learnable attention sinks. It also swaps the standard residual connection for Independent Hyper-Connections. That's four distinct architecture ideas in a single model, not one or two.

The same 5.17.0 release adds Moonshot's KimiLinear, built on per-channel forget gates, and Microsoft's VibeVoice, a long-form multi-speaker text-to-speech model. Neither shares Hy4-Preview's attention stack. The release reads like three separate architecture bets landing in the same version, not one coordinated design.

The release notes describe each component but don't include benchmark comparisons against simpler MoE architectures at the same active-parameter count. So it's not yet clear whether stacking four attention tricks beats what a conventional 49B-active model does with far less engineering complexity.