A Multi-Modal Transformer Network for Action Detection
This work addresses the problem of accurate action detection in videos for applications like surveillance or education, but it is incremental as it builds on existing transformer and multi-modal approaches.
The paper tackles action detection in untrimmed videos by proposing a multi-modal transformer network with a novel attention mechanism and an algorithm to correct motion distortion from camera movement, achieving state-of-the-art results on THUMOS14 and ActivityNet benchmarks.
This paper proposes a novel multi-modal transformer network for detecting actions in untrimmed videos. To enrich the action features, our transformer network utilizes a new multi-modal attention mechanism that computes the correlations between different spatial and motion modalities combinations. Exploring such correlations for actions has not been attempted previously. To use the motion and spatial modality more effectively, we suggest an algorithm that corrects the motion distortion caused by camera movement. Such motion distortion, common in untrimmed videos, severely reduces the expressive power of motion features such as optical flow fields. Our proposed algorithm outperforms the state-of-the-art methods on two public benchmarks, THUMOS14 and ActivityNet. We also conducted comparative experiments on our new instructional activity dataset, including a large set of challenging classroom videos captured from elementary schools.