HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actions
This work addresses the problem of self-supervised learning for skeleton-based action recognition, which is incremental as it builds on prior contrastive learning methods by introducing a novel way to generate positives.
The paper tackles the challenge of learning skeleton sequence encoders without labels for action recognition by proposing HaLP, a module that hallucinates latent positives for contrastive learning, leading to consistent improvements on benchmarks like NTU-60, NTU-120, and PKU-II in tasks such as linear evaluation, transfer learning, and kNN evaluation.
Supervised learning of skeleton sequence encoders for action recognition has received significant attention in recent times. However, learning such encoders without labels continues to be a challenging problem. While prior works have shown promising results by applying contrastive learning to pose sequences, the quality of the learned representations is often observed to be closely tied to data augmentations that are used to craft the positives. However, augmenting pose sequences is a difficult task as the geometric constraints among the skeleton joints need to be enforced to make the augmentations realistic for that action. In this work, we propose a new contrastive learning approach to train models for skeleton-based action recognition without labels. Our key contribution is a simple module, HaLP - to Hallucinate Latent Positives for contrastive learning. Specifically, HaLP explores the latent space of poses in suitable directions to generate new positives. To this end, we present a novel optimization formulation to solve for the synthetic positives with an explicit control on their hardness. We propose approximations to the objective, making them solvable in closed form with minimal overhead. We show via experiments that using these generated positives within a standard contrastive learning framework leads to consistent improvements across benchmarks such as NTU-60, NTU-120, and PKU-II on tasks like linear evaluation, transfer learning, and kNN evaluation. Our code will be made available at https://github.com/anshulbshah/HaLP.