Temporal Transformer Networks with Self-Supervision for Action Recognition
This work addresses a key bottleneck in video action recognition for computer vision applications, though it appears incremental as it builds on existing transformer and self-supervision ideas.
The paper tackles the problem of limited long-range temporal and reverse motion modeling in video action recognition by introducing a Temporal Transformer Network with Self-supervision (TTSN), achieving state-of-the-art performance on datasets like HMDB51, UCF101, and Something-something V1.
In recent years, 2D Convolutional Networks-based video action recognition has encouragingly gained wide popularity; However, constrained by the lack of long-range non-linear temporal relation modeling and reverse motion information modeling, the performance of existing models is, therefore, undercut seriously. To address this urgent problem, we introduce a startling Temporal Transformer Network with Self-supervision (TTSN). Our high-performance TTSN mainly consists of a temporal transformer module and a temporal sequence self-supervision module. Concisely speaking, we utilize the efficient temporal transformer module to model the non-linear temporal dependencies among non-local frames, which significantly enhances complex motion feature representations. The temporal sequence self-supervision module we employ unprecedentedly adopts the streamlined strategy of "random batch random channel" to reverse the sequence of video frames, allowing robust extractions of motion information representation from inversed temporal dimensions and improving the generalization capability of the model. Extensive experiments on three widely used datasets (HMDB51, UCF101, and Something-something V1) have conclusively demonstrated that our proposed TTSN is promising as it successfully achieves state-of-the-art performance for action recognition.