Agnes-3.0-Flash is a 33B Apache-2.0 multimodal model where only 18 of 72 layers hold a growing KV cache.

Three of every four layers run a gated delta rule with per-layer state independent of sequence length; the fourth runs standard global attention. Across a 262,144-token window with text, image and video understanding, that means only 18 layers carry a cache scaling with context (Hugging Face). It's dense, not MoE, which the r/LocalLLaMA thread argues still makes it the best option at 32 GB of VRAM. Hybrid attention where the cache cost is structurally bounded is the shape I expect long-context local models to converge on.