ATM: Action Temporality Modeling for Video Question Answering
This addresses a key bottleneck in video understanding for AI systems, offering incremental improvements in temporality reasoning.
The paper tackles the problem of causal/temporal reasoning in video question answering by introducing Action Temporality Modeling (ATM), which improves accuracy on multiple VideoQA benchmarks through better motion representations and action-centric training.
Despite significant progress in video question answering (VideoQA), existing methods fall short of questions that require causal/temporal reasoning across frames. This can be attributed to imprecise motion representations. We introduce Action Temporality Modeling (ATM) for temporality reasoning via three-fold uniqueness: (1) rethinking the optical flow and realizing that optical flow is effective in capturing the long horizon temporality reasoning; (2) training the visual-text embedding by contrastive learning in an action-centric manner, leading to better action representations in both vision and text modalities; and (3) preventing the model from answering the question given the shuffled video in the fine-tuning stage, to avoid spurious correlation between appearance and motion and hence ensure faithful temporality reasoning. In the experiments, we show that ATM outperforms previous approaches in terms of the accuracy on multiple VideoQAs and exhibits better true temporality reasoning ability.