What Can Simple Arithmetic Operations Do for Temporal Modeling?
This work addresses efficient temporal modeling for video understanding, offering a plug-and-play module that improves performance with low computational cost, though it appears incremental as it builds on existing feature extraction methods.
The paper tackles temporal modeling in videos by proposing an Arithmetic Temporal Module (ATM) that uses simple arithmetic operations on frame features, achieving top-1 accuracies of 65.6%, 74.6%, and 89.4% on Something-Something V1, V2, and Kinetics-400 benchmarks.
Temporal modeling plays a crucial role in understanding video content. To tackle this problem, previous studies built complicated temporal relations through time sequence thanks to the development of computationally powerful devices. In this work, we explore the potential of four simple arithmetic operations for temporal modeling. Specifically, we first capture auxiliary temporal cues by computing addition, subtraction, multiplication, and division between pairs of extracted frame features. Then, we extract corresponding features from these cues to benefit the original temporal-irrespective domain. We term such a simple pipeline as an Arithmetic Temporal Module (ATM), which operates on the stem of a visual backbone with a plug-and-play style. We conduct comprehensive ablation studies on the instantiation of ATMs and demonstrate that this module provides powerful temporal modeling capability at a low computational cost. Moreover, the ATM is compatible with both CNNs- and ViTs-based architectures. Our results show that ATM achieves superior performance over several popular video benchmarks. Specifically, on Something-Something V1, V2 and Kinetics-400, we reach top-1 accuracy of 65.6%, 74.6%, and 89.4% respectively. The code is available at https://github.com/whwu95/ATM.