3D Hand Pose Estimation in Everyday Egocentric Images
This work solves the problem of accurate 3D hand pose estimation in real-world egocentric scenarios for applications like AR/VR and robotics, though it is incremental as it builds on existing practices with systematic improvements.
The paper tackles 3D hand pose estimation in everyday egocentric images by addressing challenges like poor visual signal, perspective distortion, and lack of 3D annotations, resulting in a system called WildHands that beats past methods by 7.4% to 66% on zero-shot evaluations across diverse datasets and outperforms others in system-level comparisons while being smaller and trained on less data.
3D hand pose estimation in everyday egocentric images is challenging for several reasons: poor visual signal (occlusion from the object of interaction, low resolution & motion blur), large perspective distortion (hands are close to the camera), and lack of 3D annotations outside of controlled settings. While existing methods often use hand crops as input to focus on fine-grained visual information to deal with poor visual signal, the challenges arising from perspective distortion and lack of 3D annotations in the wild have not been systematically studied. We focus on this gap and explore the impact of different practices, i.e. crops as input, incorporating camera information, auxiliary supervision, scaling up datasets. We provide several insights that are applicable to both convolutional and transformer models leading to better performance. Based on our findings, we also present WildHands, a system for 3D hand pose estimation in everyday egocentric images. Zero-shot evaluation on 4 diverse datasets (H2O, AssemblyHands, Epic-Kitchens, Ego-Exo4D) demonstrate the effectiveness of our approach across 2D and 3D metrics, where we beat past methods by 7.4% - 66%. In system level comparisons, WildHands achieves the best 3D hand pose on ARCTIC egocentric split, outperforms FrankMocap across all metrics and HaMeR on 3 out of 6 metrics while being 10x smaller and trained on 5x less data.