PAFUSE: Part-based Diffusion for 3D Whole-Body Pose Estimation
This work solves the challenge of accurately estimating fine-grained 3D poses for whole-body applications, such as in computer vision or human-computer interaction, and it is incremental by building on diffusion models and part-based representations.
The paper tackles the problem of 3D whole-body pose estimation by addressing scale and deformability variance across body parts, and it achieves state-of-the-art performance on the H3WB dataset with significant improvements over existing methods.
We introduce a novel approach for 3D whole-body pose estimation, addressing the challenge of scale -- and deformability -- variance across body parts brought by the challenge of extending the 17 major joints on the human body to fine-grained keypoints on the face and hands. In addition to addressing the challenge of exploiting motion in unevenly sampled data, we combine stable diffusion to a hierarchical part representation which predicts the relative locations of fine-grained keypoints within each part (e.g., face) with respect to the part's local reference frame. On the H3WB dataset, our method greatly outperforms the current state of the art, which fails to exploit the temporal information. We also show considerable improvements compared to other spatiotemporal 3D human-pose estimation approaches that fail to account for the body part specificities. Code is available at https://github.com/valeoai/PAFUSE.