transformers v5.16.0 adds Qwen4-Exp, the first hybrid architecture combining linear and sparse attention

GitHub·medium signalXBlueskyLinkedInCopy link

Released 2026-08-26, v5.16.0 adds Qwen4-Exp with three new components: GatedResidual, which mixes multiple residual streams with elementwise gating before each attention and MoE block; Qwen Sparse Attention, which scores compressed key blocks and selects contiguous token blocks while leaving the trailing block uncompressed; and Per-Layer Embedding, which enriches selected decoder layers with hashed token n-gram features. Combined with Gated DeltaNet, QSA makes Qwen4-Exp the first hybrid architecture to integrate both linear and sparse attention. The block-level selection is aimed squarely at long-context inference cost, which is the binding constraint for agent workloads.