STARS: Self-supervised Tuning for 3D Action Recognition in Skeleton Sequences
This work addresses a specific issue in skeleton-based action recognition for computer vision applications, offering an incremental improvement over existing self-supervised methods.
The paper tackles the problem of self-supervised pretraining for 3D skeleton-based action recognition, showing that masked prediction methods lack well-separated clusters and generalize poorly in few-shot settings, and proposes STARS, which combines masked prediction with nearest-neighbor contrastive tuning to achieve state-of-the-art results on benchmarks like NTU-60, NTU-120, and PKU-MMD, with significant improvements in few-shot scenarios.
Self-supervised pretraining methods with masked prediction demonstrate remarkable within-dataset performance in skeleton-based action recognition. However, we show that, unlike contrastive learning approaches, they do not produce well-separated clusters. Additionally, these methods struggle with generalization in few-shot settings. To address these issues, we propose Self-supervised Tuning for 3D Action Recognition in Skeleton sequences (STARS). Specifically, STARS first uses a masked prediction stage using an encoder-decoder architecture. It then employs nearest-neighbor contrastive learning to partially tune the weights of the encoder, enhancing the formation of semantic clusters for different actions. By tuning the encoder for a few epochs, and without using hand-crafted data augmentations, STARS achieves state-of-the-art self-supervised results in various benchmarks, including NTU-60, NTU-120, and PKU-MMD. In addition, STARS exhibits significantly better results than masked prediction models in few-shot settings, where the model has not seen the actions throughout pretraining. Project page: https://soroushmehraban.github.io/stars/