MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition
This addresses the problem of efficient and accurate video understanding for applications like surveillance and entertainment, offering a novel approach that avoids heavy optical flow computation.
The paper tackles video action recognition by proposing MM-ViT, a transformer-based method that operates in the compressed video domain using I-frames, motion vectors, residuals, and audio, achieving state-of-the-art efficiency and accuracy on benchmarks like UCF-101, Something-Something-v2, and Kinetics-600.
This paper presents a pure transformer-based approach, dubbed the Multi-Modal Video Transformer (MM-ViT), for video action recognition. Different from other schemes which solely utilize the decoded RGB frames, MM-ViT operates exclusively in the compressed video domain and exploits all readily available modalities, i.e., I-frames, motion vectors, residuals and audio waveform. In order to handle the large number of spatiotemporal tokens extracted from multiple modalities, we develop several scalable model variants which factorize self-attention across the space, time and modality dimensions. In addition, to further explore the rich inter-modal interactions and their effects, we develop and compare three distinct cross-modal attention mechanisms that can be seamlessly integrated into the transformer building block. Extensive experiments on three public action recognition benchmarks (UCF-101, Something-Something-v2, Kinetics-600) demonstrate that MM-ViT outperforms the state-of-the-art video transformers in both efficiency and accuracy, and performs better or equally well to the state-of-the-art CNN counterparts with computationally-heavy optical flow.