Multi-Scale Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition
This work addresses the need for efficient and accurate action recognition from skeleton data, which is important for applications like surveillance and human-computer interaction, and it is incremental as it builds on existing neural network approaches with specific enhancements.
The paper tackles the problem of inefficient deep networks for skeleton-based action recognition by proposing a multi-scale semantics-guided neural network (MS-SGN) that incorporates high-level semantics and hierarchical joint relationships, achieving state-of-the-art performance with a model size an order of magnitude smaller than previous methods on NTU60, NTU120, and SYSU datasets.
Skeleton data is of low dimension. However, there is a trend of using very deep and complicated feedforward neural networks to model the skeleton sequence without considering the complexity in recent year. In this paper, a simple yet effective multi-scale semantics-guided neural network (MS-SGN) is proposed for skeleton-based action recognition. We explicitly introduce the high level semantics of joints (joint type and frame index) into the network to enhance the feature representation capability of joints. Moreover, a multi-scale strategy is proposed to be robust to the temporal scale variations. In addition, we exploit the relationship of joints hierarchically through two modules, i.e., a joint-level module for modeling the correlations of joints in the same frame and a frame-level module for modeling the temporal dependencies of frames. With an order of magnitude smaller model size than most previous methods, MSSGN achieves the state-of-the-art performance on the NTU60, NTU120, and SYSU datasets.