HMP: Hand Motion Priors for Pose and Shape Estimation from Video
This work addresses the need for accurate 3D hand pose estimation for understanding human interactions, but it is incremental as it builds on existing video-based methods by incorporating a motion prior.
The paper tackles the problem of 3D hand pose and shape estimation from video, which is challenging due to articulation, occlusions, and rapid motions, by developing a generative hand motion prior trained on the AMASS dataset and using it in a latent optimization approach, resulting in enhanced performance, especially in occluded scenarios, with stable and temporally consistent outcomes that surpass single-frame methods.
Understanding how humans interact with the world necessitates accurate 3D hand pose estimation, a task complicated by the hand's high degree of articulation, frequent occlusions, self-occlusions, and rapid motions. While most existing methods rely on single-image inputs, videos have useful cues to address aforementioned issues. However, existing video-based 3D hand datasets are insufficient for training feedforward models to generalize to in-the-wild scenarios. On the other hand, we have access to large human motion capture datasets which also include hand motions, e.g. AMASS. Therefore, we develop a generative motion prior specific for hands, trained on the AMASS dataset which features diverse and high-quality hand motions. This motion prior is then employed for video-based 3D hand motion estimation following a latent optimization approach. Our integration of a robust motion prior significantly enhances performance, especially in occluded scenarios. It produces stable, temporally consistent results that surpass conventional single-frame methods. We demonstrate our method's efficacy via qualitative and quantitative evaluations on the HO3D and DexYCB datasets, with special emphasis on an occlusion-focused subset of HO3D. Code is available at https://hmp.is.tue.mpg.de