TAM: Temporal Adaptive Module for Video Recognition
This work addresses the problem of efficient temporal modeling in video recognition for computer vision researchers, offering an incremental improvement by integrating a modular block into existing 2D CNNs.
The paper tackles the challenge of capturing complex temporal dynamics in video data by introducing a Temporal Adaptive Module (TAM) that generates video-specific temporal kernels, achieving state-of-the-art performance on Kinetics-400 and Something-Something datasets with minimal extra computational cost.
Video data is with complex temporal dynamics due to various factors such as camera motion, speed variation, and different activities. To effectively capture this diverse motion pattern, this paper presents a new temporal adaptive module ({\bf TAM}) to generate video-specific temporal kernels based on its own feature map. TAM proposes a unique two-level adaptive modeling scheme by decoupling the dynamic kernel into a location sensitive importance map and a location invariant aggregation weight. The importance map is learned in a local temporal window to capture short-term information, while the aggregation weight is generated from a global view with a focus on long-term structure. TAM is a modular block and could be integrated into 2D CNNs to yield a powerful video architecture (TANet) with a very small extra computational cost. The extensive experiments on Kinetics-400 and Something-Something datasets demonstrate that our TAM outperforms other temporal modeling methods consistently, and achieves the state-of-the-art performance under the similar complexity. The code is available at \url{ https://github.com/liu-zhy/temporal-adaptive-module}.