Masked Motion Predictors are Strong 3D Action Representation Learners
This work addresses the problem of improving 3D action recognition for researchers and practitioners by introducing a novel pre-training method that enhances feature representation, though it is incremental as it builds on existing self-supervised approaches.
The paper tackles the challenge of limited supervised data in 3D human action recognition by proposing a self-supervised pre-training framework called Masked Motion Prediction (MAMP), which focuses on explicit contextual motion modeling instead of masked self-component reconstruction, achieving state-of-the-art results on datasets like NTU-60, NTU-120, and PKU-MMD.
In 3D human action recognition, limited supervised data makes it challenging to fully tap into the modeling potential of powerful networks such as transformers. As a result, researchers have been actively investigating effective self-supervised pre-training strategies. In this work, we show that instead of following the prevalent pretext task to perform masked self-component reconstruction in human joints, explicit contextual motion modeling is key to the success of learning effective feature representation for 3D action recognition. Formally, we propose the Masked Motion Prediction (MAMP) framework. To be specific, the proposed MAMP takes as input the masked spatio-temporal skeleton sequence and predicts the corresponding temporal motion of the masked human joints. Considering the high temporal redundancy of the skeleton sequence, in our MAMP, the motion information also acts as an empirical semantic richness prior that guide the masking process, promoting better attention to semantically rich temporal regions. Extensive experiments on NTU-60, NTU-120, and PKU-MMD datasets show that the proposed MAMP pre-training substantially improves the performance of the adopted vanilla transformer, achieving state-of-the-art results without bells and whistles. The source code of our MAMP is available at https://github.com/maoyunyao/MAMP.