Towards To-a-T Spatio-Temporal Focus for Skeleton-Based Action Recognition
This work addresses the challenge of modeling high-order spatio-temporal dependencies in action recognition for applications like human-computer interaction, but it is incremental as it builds on existing GCN methods with novel attention mechanisms.
The paper tackles the problem of skeleton-based action recognition by proposing the To-a-T Spatio-Temporal Focus (STF) framework, which uses spatio-temporal gradients and explicit loss terms to guide attention, resulting in state-of-the-art performance on multiple datasets including NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400 across all 15 settings.
Graph Convolutional Networks (GCNs) have been widely used to model the high-order dynamic dependencies for skeleton-based action recognition. Most existing approaches do not explicitly embed the high-order spatio-temporal importance to joints' spatial connection topology and intensity, and they do not have direct objectives on their attention module to jointly learn when and where to focus on in the action sequence. To address these problems, we propose the To-a-T Spatio-Temporal Focus (STF), a skeleton-based action recognition framework that utilizes the spatio-temporal gradient to focus on relevant spatio-temporal features. We first propose the STF modules with learnable gradient-enforced and instance-dependent adjacency matrices to model the high-order spatio-temporal dynamics. Second, we propose three loss terms defined on the gradient-based spatio-temporal focus to explicitly guide the classifier when and where to look at, distinguish confusing classes, and optimize the stacked STF modules. STF outperforms the state-of-the-art methods on the NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400 datasets in all 15 settings over different views, subjects, setups, and input modalities, and STF also shows better accuracy on scarce data and dataset shifting settings.