STEP: Simultaneous Tracking and Estimation of Pose for Animals and Humans
This work addresses the challenge of efficient and accurate pose estimation in videos for applications like action recognition and behavioral analysis, representing a novel integration of tracking and pose estimation but with incremental methodological improvements.
The paper tackles the problem of simultaneous tracking and pose estimation for animals and humans by introducing STEP, a Transformer-based framework that eliminates the need for per-frame detections, resulting in improved inference efficiency and superior performance on diverse datasets.
We introduce STEP, a novel framework utilizing Transformer-based discriminative model prediction for simultaneous tracking and estimation of pose across diverse animal species and humans. We are inspired by the fact that the human brain exploits spatiotemporal continuity and performs concurrent localization and pose estimation despite the specialization of brain areas for form and motion processing. Traditional discriminative models typically require predefined target states for determining model weights, a challenge we address through Gaussian Map Soft Prediction (GMSP) and Offset Map Regression Adapter (OMRA) Modules. These modules remove the necessity of keypoint target states as input, streamlining the process. Our method starts with a known target state in the initial frame of a given video sequence. It then seamlessly tracks the target and estimates keypoints of anatomical importance as output for subsequent frames. Unlike prevalent top-down pose estimation methods, our approach doesn't rely on per-frame target detections due to its tracking capability. This facilitates a significant advancement in inference efficiency and potential applications. We train and validate our approach on datasets encompassing diverse species. Our experiments demonstrate superior results compared to existing methods, opening doors to various applications, including but not limited to action recognition and behavioral analysis.