On The Application of Linear Attention in Multimodal Transformers
For researchers building scalable vision-language models, this work demonstrates that Linear Attention can reduce computational overhead without sacrificing performance, addressing a key bottleneck in multimodal Transformers.
The paper investigates Linear Attention (LA) as a replacement for quadratic softmax attention in multimodal Transformers, achieving linear complexity while preserving competitive performance. On ViT models trained on LAION-400M, LA matches scaling laws of standard attention and yields significant computational savings.
Multimodal Transformers serve as the backbone for state-of-the-art vision-language models, yet their quadratic attention complexity remains a critical barrier to scalability. In this work, we investigate the viability of Linear Attention (LA) as a high-efficiency alternative within multimodal frameworks. By integrating LA, we reduce the computational overhead from quadratic to linear relative to sequence length while preserving competitive performance. We evaluate our approach across ViT-S/16, ViT-B/16, and ViT-L/16 architectures trained on the LAION-400M dataset, with validation focused on ImageNet-21K zero-shot accuracy. Our systematic evaluation demonstrates that Linear Attention not only yields significant computational savings but also adheres to the same scaling laws as standard softmax attention. These findings position Linear Attention as a robust, scalable solution for next-generation multimodal Transformers tasked with processing increasingly large and complex datasets.