Snakes and Ladders: Two Steps Up for VideoMamba
This work addresses efficiency issues in video analysis for researchers and practitioners, offering incremental improvements to existing Mamba-based models.
The paper tackled the computational burden of self-attention in video understanding by identifying limitations in Mamba's token processing and proposing VideoMambaPro, which improved top-1 accuracy by 1.6-2.8% on Kinetics-400 and 1.1-1.9% on Something-Something V2 over VideoMamba.
Video understanding requires the extraction of rich spatio-temporal representations, which transformer models achieve through self-attention. Unfortunately, self-attention poses a computational burden. In NLP, Mamba has surfaced as an efficient alternative for transformers. However, Mamba's successes do not trivially extend to vision tasks, including those in video analysis. In this paper, we theoretically analyze the differences between self-attention and Mamba. We identify two limitations in Mamba's token processing: historical decay and element contradiction. We propose VideoMambaPro (VMP) that solves the identified limitations by adding masked backward computation and elemental residual connections to a VideoMamba backbone. Differently sized VideoMambaPro models surpass VideoMamba by 1.6-2.8% and 1.1-1.9% top-1 on Kinetics-400 and Something-Something V2, respectively. Even without extensive pre-training, our models present an increasingly attractive and efficient alternative to current transformer models. Moreover, our two solutions are orthogonal to recent advances in Vision Mamba models, and are likely to provide further improvements in future models.