Pose And Joint-Aware Action Recognition
This work addresses the problem of improving action recognition accuracy for computer vision applications by focusing on joint-based modalities, offering incremental advancements over existing methods.
The paper tackles joint-based action recognition by introducing a model that extracts motion features from individual joints, re-weights discriminative joints, and uses a joint-contrastive loss and geometry-aware data augmentation, achieving large improvements over state-of-the-art methods on datasets like JHMDB, HMDB, Charades, and AVA, with additional gains from fusion with RGB and flow approaches.
Recent progress on action recognition has mainly focused on RGB and optical flow features. In this paper, we approach the problem of joint-based action recognition. Unlike other modalities, constellation of joints and their motion generate models with succinct human motion information for activity recognition. We present a new model for joint-based action recognition, which first extracts motion features from each joint separately through a shared motion encoder before performing collective reasoning. Our joint selector module re-weights the joint information to select the most discriminative joints for the task. We also propose a novel joint-contrastive loss that pulls together groups of joint features which convey the same action. We strengthen the joint-based representations by using a geometry-aware data augmentation technique which jitters pose heatmaps while retaining the dynamics of the action. We show large improvements over the current state-of-the-art joint-based approaches on JHMDB, HMDB, Charades, AVA action recognition datasets. A late fusion with RGB and Flow-based approaches yields additional improvements. Our model also outperforms the existing baseline on Mimetics, a dataset with out-of-context actions.