Ordered Pooling of Optical Flow Sequences for Action Recognition
This addresses the problem of efficient action recognition for video analysis, but it is incremental as it builds on existing summarization techniques.
The paper tackles the computational expense of training CNNs on long videos by proposing an ordered representation of optical flow frames as a compact alternative to RGB frames, achieving significant improvements over RGB and comparable accuracy to state-of-the-art on UCF101 and HMDB datasets.
Training of Convolutional Neural Networks (CNNs) on long video sequences is computationally expensive due to the substantial memory requirements and the massive number of parameters that deep architectures demand. Early fusion of video frames is thus a standard technique, in which several consecutive frames are first agglomerated into a compact representation, and then fed into the CNN as an input sample. For this purpose, a summarization approach that represents a set of consecutive RGB frames by a single dynamic image to capture pixel dynamics is proposed recently. In this paper, we introduce a novel ordered representation of consecutive optical flow frames as an alternative and argue that this representation captures the action dynamics more effectively than RGB frames. We provide intuitions on why such a representation is better for action recognition. We validate our claims on standard benchmark datasets and demonstrate that using summaries of flow images lead to significant improvements over RGB frames while achieving accuracy comparable to the state-of-the-art on UCF101 and HMDB datasets.