Recognition of Visually Perceived Compositional Human Actions by Multiple Spatio-Temporal Scales Recurrent Neural Networks
This work addresses action recognition in computer vision, which is important for applications like surveillance and human-computer interaction, but it appears incremental as it builds on existing CNN and RNN approaches.
The paper tackles the problem of recognizing visually perceived human actions by proposing a multiple spatio-temporal scales recurrent neural network (MSTRNN) model, which introduces multiple timescale recurrent dynamics to a conventional CNN, and it evaluates the model on three human action video datasets, comparing performance with other deep learning models and analyzing internal representations to clarify functional hierarchy development.
The current paper proposes a novel neural network model for recognizing visually perceived human actions. The proposed multiple spatio-temporal scales recurrent neural network (MSTRNN) model is derived by introducing multiple timescale recurrent dynamics to the conventional convolutional neural network model. One of the essential characteristics of the MSTRNN is that its architecture imposes both spatial and temporal constraints simultaneously on the neural activity which vary in multiple scales among different layers. As suggested by the principle of the upward and downward causation, it is assumed that the network can develop meaningful structures such as functional hierarchy by taking advantage of such constraints during the course of learning. To evaluate the characteristics of the model, the current study uses three types of human action video dataset consisting of different types of primitive actions and different levels of compositionality on them. The performance of the MSTRNN in testing with these dataset is compared with the ones by other representative deep learning models used in the field. The analysis of the internal representation obtained through the learning with the dataset clarifies what sorts of functional hierarchy can be developed by extracting the essential compositionality underlying the dataset.