DeltaDorsal: Enhancing Hand Pose Estimation with Dorsal Features in Egocentric Views
This addresses the challenge of reliable hand pose estimation for XR device users in occluded scenarios, representing a strong domain-specific improvement.
The paper tackles the problem of finger occlusions in egocentric hand pose estimation by leveraging dorsal hand skin deformation features, achieving an 18% reduction in Mean Per Joint Angle Error in self-occluded scenarios compared to state-of-the-art methods.
The proliferation of XR devices has made egocentric hand pose estimation a vital task, yet this perspective is inherently challenged by frequent finger occlusions. To address this, we propose a novel approach that leverages the rich information in dorsal hand skin deformation, unlocked by recent advances in dense visual featurizers. We introduce a dual-stream delta encoder that learns pose by contrasting features from a dynamic hand with a baseline relaxed position. Our evaluation demonstrates that, using only cropped dorsal images, our method reduces the Mean Per Joint Angle Error (MPJAE) by 18% in self-occluded scenarios (fingers >=50% occluded) compared to state-of-the-art techniques that depend on the whole hand's geometry and large model backbones. Consequently, our method not only enhances the reliability of downstream tasks like index finger pinch and tap estimation in occluded scenarios but also unlocks new interaction paradigms, such as detecting isometric force for a surface "click" without visible movement while minimizing model size.