Gate-Shift Networks for Video Action Recognition
This work addresses the problem of efficient video action recognition for computer vision applications, offering a novel method that reduces model complexity while maintaining performance.
The paper tackles the challenge of training deep 3D CNNs for video action recognition due to high computational costs and data requirements by introducing Gate-Shift Networks, which use a lightweight module to adaptively route features through time with minimal overhead, achieving state-of-the-art results on Something Something-V1 and Diving48 datasets and competitive results on EPIC-Kitchens.
Deep 3D CNNs for video action recognition are designed to learn powerful representations in the joint spatio-temporal feature space. In practice however, because of the large number of parameters and computations involved, they may under-perform in the lack of sufficiently large datasets for training them at scale. In this paper we introduce spatial gating in spatial-temporal decomposition of 3D kernels. We implement this concept with Gate-Shift Module (GSM). GSM is lightweight and turns a 2D-CNN into a highly efficient spatio-temporal feature extractor. With GSM plugged in, a 2D-CNN learns to adaptively route features through time and combine them, at almost no additional parameters and computational overhead. We perform an extensive evaluation of the proposed module to study its effectiveness in video action recognition, achieving state-of-the-art results on Something Something-V1 and Diving48 datasets, and obtaining competitive results on EPIC-Kitchens with far less model complexity.