VideoLightFormer: Lightweight Action Recognition using Transformers
This work addresses the problem of real-world efficiency in video action recognition for applications like robotics or surveillance, though it is incremental as it builds on existing transformer and convolutional methods.
The paper tackled efficient video action recognition by proposing VideoLightFormer, a lightweight transformer-based architecture that maintains full spatiotemporal structure, achieving a better mix of efficiency and accuracy on EPIC-KITCHENS-100 and Something-Something-V2 datasets compared to most state-of-the-art models.
Efficient video action recognition remains a challenging problem. One large model after another takes the place of the state-of-the-art on the Kinetics dataset, but real-world efficiency evaluations are often lacking. In this work, we fill this gap and investigate the use of transformers for efficient action recognition. We propose a novel, lightweight action recognition architecture, VideoLightFormer. In a factorized fashion, we carefully extend the 2D convolutional Temporal Segment Network with transformers, while maintaining spatial and temporal video structure throughout the entire model. Existing methods often resort to one of the two extremes, where they either apply huge transformers to video features, or minimal transformers on highly pooled video features. Our method differs from them by keeping the transformer models small, but leveraging full spatiotemporal feature structure. We evaluate VideoLightFormer in a high-efficiency setting on the temporally-demanding EPIC-KITCHENS-100 and Something-Something-V2 (SSV2) datasets and find that it achieves a better mix of efficiency and accuracy than existing state-of-the-art models, apart from the Temporal Shift Module on SSV2.