Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition
This addresses the problem of high computational demands and overfitting in 3D CNNs for researchers and practitioners in video analysis, though it is incremental as it builds on existing 3D CNN frameworks.
The paper tackles the computational inefficiency and feature learning limitations of conventional 3D CNNs for human action recognition by proposing spatio-temporal STFT blocks, which reduce parameters by 3.5-4.5 times and computational costs by 1.5-1.8 times while achieving on par or better performance on seven datasets.
Conventional 3D convolutional neural networks (CNNs) are computationally expensive, memory intensive, prone to overfitting, and most importantly, there is a need to improve their feature learning capabilities. To address these issues, we propose spatio-temporal short term Fourier transform (STFT) blocks, a new class of convolutional blocks that can serve as an alternative to the 3D convolutional layer and its variants in 3D CNNs. An STFT block consists of non-trainable convolution layers that capture spatially and/or temporally local Fourier information using a STFT kernel at multiple low frequency points, followed by a set of trainable linear weights for learning channel correlations. The STFT blocks significantly reduce the space-time complexity in 3D CNNs. In general, they use 3.5 to 4.5 times less parameters and 1.5 to 1.8 times less computational costs when compared to the state-of-the-art methods. Furthermore, their feature learning capabilities are significantly better than the conventional 3D convolutional layer and its variants. Our extensive evaluation on seven action recognition datasets, including Something-something v1 and v2, Jester, Diving-48, Kinetics-400, UCF 101, and HMDB 51, demonstrate that STFT blocks based 3D CNNs achieve on par or even better performance compared to the state-of-the-art methods.