Do VLMs Need Vision Transformers? State Space Models Match ViT Encoders with Lower Memory Cost

arXiv 2603.19209·medium signal

Researchers systematically evaluate whether Mamba-class state space models can replace frozen Vision Transformers as backbone encoders in large VLMs, finding competitive benchmark performance. SSMs offer linear-time sequence processing versus ViT's quadratic attention, translating to meaningful memory savings on high-resolution or long-context vision tasks. Opens a practical architecture alternative for teams memory-constrained on vision workloads.