Gate-Shift-Pose: Enhancing Action Recognition in Sports with Skeleton Information
This work addresses action recognition in sports, specifically for figure skating safety, and is incremental as it enhances an existing network with multimodal fusion.
The paper tackles athlete fall classification in figure skating by integrating skeleton pose data with RGB frames using Gate-Shift-Pose, achieving up to 40% accuracy improvement over RGB-only baselines and a top accuracy of 98.08% with ResNet50.
This paper introduces Gate-Shift-Pose, an enhanced version of Gate-Shift-Fuse networks, designed for athlete fall classification in figure skating by integrating skeleton pose data alongside RGB frames. We evaluate two fusion strategies: early-fusion, which combines RGB frames with Gaussian heatmaps of pose keypoints at the input stage, and late-fusion, which employs a multi-stream architecture with attention mechanisms to combine RGB and pose features. Experiments on the FR-FS dataset demonstrate that Gate-Shift-Pose significantly outperforms the RGB-only baseline, improving accuracy by up to 40% with ResNet18 and 20% with ResNet50. Early-fusion achieves the highest accuracy (98.08%) with ResNet50, leveraging the model's capacity for effective multimodal integration, while late-fusion is better suited for lighter backbones like ResNet18. These results highlight the potential of multimodal architectures for sports action recognition and the critical role of skeleton pose information in capturing complex motion patterns. Visit the project page at https://edowhite.github.io/Gate-Shift-Pose