Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers
This addresses computational bottlenecks for researchers and practitioners working with long video understanding, though it appears incremental as it combines existing Mamba and Transformer components.
The paper tackles the problem of transformer-based models struggling with hour-long videos due to quadratic complexity, and introduces VAMBA, a hybrid Mamba-Transformer model that achieves a 4.3% accuracy improvement on the LVBench benchmark while reducing GPU memory usage by at least 50% and nearly doubling training speed.
State-of-the-art transformer-based large multimodal models (LMMs) struggle to handle hour-long video inputs due to the quadratic complexity of the causal self-attention operations, leading to high computational costs during training and inference. Existing token compression-based methods reduce the number of video tokens but often incur information loss and remain inefficient for extremely long sequences. In this paper, we explore an orthogonal direction to build a hybrid Mamba-Transformer model (VAMBA) that employs Mamba-2 blocks to encode video tokens with linear complexity. Without any token reduction, VAMBA can encode more than 1024 frames (640$\times$360) on a single GPU, while transformer-based models can only encode 256 frames. On long video input, VAMBA achieves at least 50% reduction in GPU memory usage during training and inference, and nearly doubles the speed per training step compared to transformer-based LMMs. Our experimental results demonstrate that VAMBA improves accuracy by 4.3% on the challenging hour-long video understanding benchmark LVBench over prior efficient video LMMs, and maintains strong performance on a broad spectrum of long and short video understanding tasks.