allenai/OLMo-core + Olmo-Hybrid-7B

GitHub·high signal

AI2 released Olmo Hybrid 7B on March 6 -- a fully open 7B model combining transformer attention with linear recurrent layers (Gated DeltaNet). Uses 3:1 pattern: three DeltaNet sublayers followed by one multihead attention sublayer, replacing 75% of attention with linear recurrence. Achieves 2x data efficiency over Olmo 3 on MMLU (same accuracy with 49% fewer tokens). Trained on 6T tokens on 512 GPUs (H100s then B200s at Lambda). Releases base, SFT, DPO stages with all weights, checkpoints, train