Direction-Aware Hybrid Representation Learning for 3D Hand Pose and Shape Estimation
This work addresses the problem of accurate and stable 3D hand pose estimation for applications like motion capture, though it appears incremental as it builds on existing model-based methods with a novel feature fusion approach.
The paper tackles the challenge of 3D hand pose and shape estimation by proposing direction-aware hybrid features (DaHyF) that fuse implicit and explicit features with pixel direction information, resulting in a method that outperforms state-of-the-art by over 33% in accuracy on the FreiHAND dataset and achieves top rankings on HO3D benchmarks.
Most model-based 3D hand pose and shape estimation methods directly regress the parametric model parameters from an image to obtain 3D joints under weak supervision. However, these methods involve solving a complex optimization problem with many local minima, making training difficult. To address this challenge, we propose learning direction-aware hybrid features (DaHyF) that fuse implicit image features and explicit 2D joint coordinate features. This fusion is enhanced by the pixel direction information in the camera coordinate system to estimate pose, shape, and camera viewpoint. Our method directly predicts 3D hand poses with DaHyF representation and reduces jittering during motion capture using prediction confidence based on contrastive learning. We evaluate our method on the FreiHAND dataset and show that it outperforms existing state-of-the-art methods by more than 33% in accuracy. DaHyF also achieves the top ranking on both the HO3Dv2 and HO3Dv3 leaderboards for the metric of Mean Joint Error (after scale and translation alignment). Compared to the second-best results, the largest improvement observed is 10%. We also demonstrate its effectiveness in real-time motion capture scenarios with hand position variability, occlusion, and motion blur.