Olmo Hybrid 7B: Transformers + Linear RNNs for 2x Data Efficiency

Interconnects (Nathan Lambert)·high signal

AI2 released Olmo Hybrid 7B, a fully open model using a 3:1 DeltaNet-to-attention architecture that replaces 75% of attention with Gated DeltaNet layers. Trained on 6T tokens across 512 GPUs (H100s then B200s). Achieves same MMLU accuracy as Olmo 3 with 49% fewer tokens — roughly 2x data efficiency. Full code, training configs, and launch scripts open-sourced. Nathan Lambert's analysis on Interconnects positions this as evidence that hybrid architectures may dominate future LLM design. Interconn