Reddit
Agnes-3.0-Flash: a 33B Apache-2.0 multimodal model where only 18 of 72 layers hold a growing KV cache
Posted to Hugging Face on 2026-09-11 (62 likes in under a day) and surfaced on r/LocalLLaMA at 132 upvotes, Agnes-3.0-Flash is a hybrid-attention decoder: three of every four layers run a gated delta rule with per-layer state independent of sequence length, and the fourth runs standard global attention. That means only 18 of its 72 layers carry a KV cache that scales with context, across a 262,144-token window with text, image and video understanding. The top comment notes it would be SOTA-for-size if it were MoE; it is dense, which a reply argues still makes it the best option at 32 GB of VRAM.
↳ Follow the thread