Correlation Net: Spatiotemporal multimodal deep learning for action recognition
This work addresses action recognition for video analysis, but it appears incremental as it complements existing network fusion methods.
The paper tackled the problem of overfitting and fusion in spatiotemporal multimodal deep learning for action recognition by proposing a correlation network with Shannon fusion to capture correlations over arbitrary timestamps. The result showed enhanced accuracy on UCF-101 and HMDB-51 datasets, though no concrete numbers were provided.
This paper describes a network that captures multimodal correlations over arbitrary timestamps. The proposed scheme operates as a complementary, extended network over a multimodal convolutional neural network (CNN). Spatial and temporal streams are required for action recognition by a deep CNN, but overfitting reduction and fusing these two streams remain open problems. The existing fusion approach averages the two streams. Here we propose a correlation network with a Shannon fusion for learning a pre-trained CNN. A Long-range video may consist of spatiotemporal correlations over arbitrary times, which can be captured by forming the correlation network from simple fully connected layers. This approach was found to complement the existing network fusion methods. The importance of multimodal correlation is validated in comparison experiments on the UCF-101 and HMDB-51 datasets. The multimodal correlation enhanced the accuracy of the video recognition results.