Spatial Temporal Transformer Network for Skeleton-based Action Recognition
This work provides an incremental improvement in skeleton-based action recognition for computer vision researchers and applications.
This paper addresses the challenge of effectively encoding latent information in 3D skeleton data for human action recognition. The proposed Spatial-Temporal Transformer network (ST-TR) achieved state-of-the-art performance on the NTU-RGB+D 60 and NTU-RGB+D 120 datasets using the same input data as existing models.
Skeleton-based human action recognition has achieved a great interest in recent years, as skeleton data has been demonstrated to be robust to illumination changes, body scales, dynamic camera views, and complex background. Nevertheless, an effective encoding of the latent information underlying the 3D skeleton is still an open problem. In this work, we propose a novel Spatial-Temporal Transformer network (ST-TR) which models dependencies between joints using the Transformer self-attention operator. In our ST-TR model, a Spatial Self-Attention module (SSA) is used to understand intra-frame interactions between different body parts, and a Temporal Self-Attention module (TSA) to model inter-frame correlations. The two are combined in a two-stream network which outperforms state-of-the-art models using the same input data on both NTU-RGB+D 60 and NTU-RGB+D 120.