Learning Transferable Self-attentive Representations for Action Recognition in Untrimmed Videos with Weak Supervision
This addresses the problem of reducing annotation costs for video action recognition, though it is incremental as it builds on existing weakly supervised and transfer learning approaches.
The paper tackles action recognition in untrimmed videos with only video-level annotations, proposing a weakly supervised framework that localizes action frames using self-attention and transfers knowledge from trimmed videos, achieving improved performance on benchmark datasets like THUMOS14 and ActivityNet1.3.
Action recognition in videos has attracted a lot of attention in the past decade. In order to learn robust models, previous methods usually assume videos are trimmed as short sequences and require ground-truth annotations of each video frame/sequence, which is quite costly and time-consuming. In this paper, given only video-level annotations, we propose a novel weakly supervised framework to simultaneously locate action frames as well as recognize actions in untrimmed videos. Our proposed framework consists of two major components. First, for action frame localization, we take advantage of the self-attention mechanism to weight each frame, such that the influence of background frames can be effectively eliminated. Second, considering that there are trimmed videos publicly available and also they contain useful information to leverage, we present an additional module to transfer the knowledge from trimmed videos for improving the classification performance in untrimmed ones. Extensive experiments are conducted on two benchmark datasets (i.e., THUMOS14 and ActivityNet1.3), and experimental results clearly corroborate the efficacy of our method.