IF-TTN: Information Fused Temporal Transformation Network for Video Action Recognition
This work addresses the problem of efficient and accurate action recognition in videos for computer vision applications, with incremental improvements in speed and robustness.
The paper tackles video action recognition by proposing IF-TTN, which fuses appearance and motion features and models temporal transformations, achieving state-of-the-art results on UCF101 and HMDB51 datasets with a 10x faster testing speed when using motion vectors instead of optical flow.
Effective spatiotemporal feature representation is crucial to the video-based action recognition task. Focusing on discriminate spatiotemporal feature learning, we propose Information Fused Temporal Transformation Network (IF-TTN) for action recognition on top of popular Temporal Segment Network (TSN) framework. In the network, Information Fusion Module (IFM) is designed to fuse the appearance and motion features at multiple ConvNet levels for each video snippet, forming a short-term video descriptor. With fused features as inputs, Temporal Transformation Networks (TTN) are employed to model middle-term temporal transformation between the neighboring snippets following a sequential order. As TSN itself depicts long-term temporal structure by segmental consensus, the proposed network comprehensively considers multiple granularity temporal features. Our IF-TTN achieves the state-of-the-art results on two most popular action recognition datasets: UCF101 and HMDB51. Empirical investigation reveals that our architecture is robust to the input motion map quality. Replacing optical flow with the motion vectors from compressed video stream, the performance is still comparable to the flow-based methods while the testing speed is 10x faster.