Stacked Temporal Attention: Improving First-person Action Recognition by Emphasizing Discriminative Clips
This work addresses the problem of noisy frames in first-person videos for action recognition, offering an incremental improvement over previous temporal attention methods by incorporating global context.
The paper tackles the challenge of first-person action recognition by proposing a Stacked Temporal Attention Module (STAM) that emphasizes discriminative clips using global context, resulting in improved performance across various datasets when integrated with existing backbones.
First-person action recognition is a challenging task in video understanding. Because of strong ego-motion and a limited field of view, many backgrounds or noisy frames in a first-person video can distract an action recognition model during its learning process. To encode more discriminative features, the model needs to have the ability to focus on the most relevant part of the video for action recognition. Previous works explored to address this problem by applying temporal attention but failed to consider the global context of the full video, which is critical for determining the relatively significant parts. In this work, we propose a simple yet effective Stacked Temporal Attention Module (STAM) to compute temporal attention based on the global knowledge across clips for emphasizing the most discriminative features. We achieve this by stacking multiple self-attention layers. Instead of naive stacking, which is experimentally proven to be ineffective, we carefully design the input to each self-attention layer so that both the local and global context of the video is considered during generating the temporal attention weights. Experiments demonstrate that our proposed STAM can be built on top of most existing backbones and boost the performance in various datasets.