Temporal-Aware Self-Supervised Learning for 3D Hand Pose and Mesh Estimation in Videos
This work addresses the problem of limited 3D hand pose datasets for researchers and developers by enabling training without explicit 3D annotations, offering a significant step towards reducing annotation burden.
This paper proposes a self-supervised learning framework, TASSN, for 3D hand pose and mesh estimation from RGB videos using only 2D keypoint annotations. By enforcing temporal consistency constraints, TASSN achieves 3D estimation accuracy on par with state-of-the-art models that are trained with explicit 3D annotations.
Estimating 3D hand pose directly from RGB imagesis challenging but has gained steady progress recently bytraining deep models with annotated 3D poses. Howeverannotating 3D poses is difficult and as such only a few 3Dhand pose datasets are available, all with limited samplesizes. In this study, we propose a new framework of training3D pose estimation models from RGB images without usingexplicit 3D annotations, i.e., trained with only 2D informa-tion. Our framework is motivated by two observations: 1)Videos provide richer information for estimating 3D posesas opposed to static images; 2) Estimated 3D poses oughtto be consistent whether the videos are viewed in the for-ward order or reverse order. We leverage these two obser-vations to develop a self-supervised learning model calledtemporal-aware self-supervised network (TASSN). By en-forcing temporal consistency constraints, TASSN learns 3Dhand poses and meshes from videos with only 2D keypointposition annotations. Experiments show that our modelachieves surprisingly good results, with 3D estimation ac-curacy on par with the state-of-the-art models trained with3D annotations, highlighting the benefit of the temporalconsistency in constraining 3D prediction models.