Gesture-Aware Pretraining and Token Fusion for 3D Hand Pose Estimation
This work addresses a domain-specific problem for AR/VR and human-computer interaction applications, with incremental improvements in accuracy.
The paper tackles 3D hand pose estimation from monocular RGB images by leveraging gesture semantics as an inductive bias, resulting in improved accuracy over the state-of-the-art EANet baseline on the InterHand2.6M dataset.
Estimating 3D hand pose from monocular RGB images is fundamental for applications in AR/VR, human-computer interaction, and sign language understanding. In this work we focus on a scenario where a discrete set of gesture labels is available and show that gesture semantics can serve as a powerful inductive bias for 3D pose estimation. We present a two-stage framework: gesture-aware pretraining that learns an informative embedding space using coarse and fine gesture labels from InterHand2.6M, followed by a per-joint token Transformer guided by gesture embeddings as intermediate representations for final regression of MANO hand parameters. Training is driven by a layered objective over parameters, joints, and structural constraints. Experiments on InterHand2.6M demonstrate that gesture-aware pretraining consistently improves single-hand accuracy over the state-of-the-art EANet baseline, and that the benefit transfers across architectures without any modification.