Deep Local Video Feature for Action Recognition
This addresses the challenge of GPU memory limitations in video analysis for action recognition, but it is incremental as it builds on existing sampling methods.
The paper tackles the problem of representing entire videos for action recognition by proposing to treat deep networks trained on local inputs as local feature extractors, aggregating them into global features, and shows that maximum pooling on sparsely sampled features leads to significant performance improvement on HMDB51 and UCF101 datasets.
We investigate the problem of representing an entire video using CNN features for human action recognition. Currently, limited by GPU memory, we have not been able to feed a whole video into CNN/RNNs for end-to-end learning. A common practice is to use sampled frames as inputs and video labels as supervision. One major problem of this popular approach is that the local samples may not contain the information indicated by global labels. To deal with this problem, we propose to treat the deep networks trained on local inputs as local feature extractors. After extracting local features, we aggregate them into global features and train another mapping function on the same training data to map the global features into global labels. We study a set of problems regarding this new type of local features such as how to aggregate them into global features. Experimental results on HMDB51 and UCF101 datasets show that, for these new local features, a simple maximum pooling on the sparsely sampled features lead to significant performance improvement.