TSI: Temporal Saliency Integration for Video Action Recognition
This work addresses a domain-specific challenge in video action recognition by improving motion feature extraction, though it appears incremental as it builds on existing methods with novel modules.
The paper tackles the problem of efficient spatiotemporal modeling for video action recognition by proposing a Temporal Saliency Integration (TSI) block with Salient Motion Excitation and Cross-perception Temporal Integration modules to handle various actions and reduce noise from camera movement, achieving state-of-the-art results on benchmarks like Something-Something V1 & V2, Kinetics-400, UCF-101, and HMDB-51.
Efficient spatiotemporal modeling is an important yet challenging problem for video action recognition. Existing state-of-the-art methods exploit neighboring feature differences to obtain motion clues for short-term temporal modeling with a simple convolution. However, only one local convolution is incapable of handling various kinds of actions because of the limited receptive field. Besides, action-irrelated noises brought by camera movement will also harm the quality of extracted motion features. In this paper, we propose a Temporal Saliency Integration (TSI) block, which mainly contains a Salient Motion Excitation (SME) module and a Cross-perception Temporal Integration (CTI) module. Specifically, SME aims to highlight the motion-sensitive area through spatial-level local-global motion modeling, where the saliency alignment and pyramidal motion modeling are conducted successively between adjacent frames to capture motion dynamics with fewer noises caused by misaligned background. CTI is designed to perform multi-perception temporal modeling through a group of separate 1D convolutions respectively. Meanwhile, temporal interactions across different perceptions are integrated with the attention mechanism. Through these two modules, long short-term temporal relationships can be encoded efficiently by introducing limited additional parameters. Extensive experiments are conducted on several popular benchmarks (i.e., Something-Something V1 & V2, Kinetics-400, UCF-101, and HMDB-51), which demonstrate the effectiveness of our proposed method.