Semantic Adversarial Network with Multi-scale Pyramid Attention for Video Classification
This addresses the problem of expensive optical flow computation and limited detail capture in video classification for researchers and practitioners, though it is incremental as it builds on existing two-stream architectures.
The paper tackled video classification by proposing a two-stream deep framework that uses only RGB frames to capture spatial and temporal information, integrating multi-scale pyramid attention and semantic adversarial learning, achieving state-of-the-art results on public benchmarks.
Two-stream architecture have shown strong performance in video classification task. The key idea is to learn spatio-temporal features by fusing convolutional networks spatially and temporally. However, there are some problems within such architecture. First, it relies on optical flow to model temporal information, which are often expensive to compute and store. Second, it has limited ability to capture details and local context information for video data. Third, it lacks explicit semantic guidance that greatly decrease the classification performance. In this paper, we proposed a new two-stream based deep framework for video classification to discover spatial and temporal information only from RGB frames, moreover, the multi-scale pyramid attention (MPA) layer and the semantic adversarial learning (SAL) module is introduced and integrated in our framework. The MPA enables the network capturing global and local feature to generate a comprehensive representation for video, and the SAL can make this representation gradually approximate to the real video semantics in an adversarial manner. Experimental results on two public benchmarks demonstrate our proposed methods achieves state-of-the-art results on standard video datasets.