Towards in-the-wild Egocentric 3D Hand-Object Pose Estimation
For researchers in egocentric 3D hand-object pose estimation, this work provides a new in-the-wild dataset and a method that significantly outperforms prior art on both in-lab and in-the-wild benchmarks.
The authors introduce EPIC-Contact, a large in-the-wild egocentric dataset with dense 3D hand-object contact annotations, and propose HOPformer, a transformer that jointly estimates bi-manual hand and object pose. HOPformer achieves 82.4% success rate on ARCTIC (+6.2 pts over SOTA) and nearly doubles success rate on EPIC-Contact while reducing contact deviation by 75%.
Estimating accurate 3D hand-object pose from in-the-wild egocentric RGB remains challenging due to severe occlusions and ambiguous contact. Existing learning-based methods often struggle to generalise to in-the-wild scenes and are limited by the scarcity of supervision. We address these issues with two contributions. First, we introduce EPIC-Contact, an in-the-wild egocentric dataset of 2.3K clips (62.3K frames) with dense, bijective 3D hand-object contact correspondences and posed meshes. Second, we propose HOPformer, an end-to-end transformer that jointly predicts bi-manual hand and object pose in a single forward pass. A cross-attention decoder conditions object features on hand priors, producing robust pose estimation. We test HOPformer on the in-lab 3D dataset, ARCTIC, as well as our newly introduced EPIC-Contact dataset. HOPformer reaches 82.4% success rate on ARCTIC (+6.2 pts over current SOTA). On EPIC-Contact, it nearly doubles the success rate while reducing contact deviation by 75%. EPIC-Contact, HOPformer code and checkpoints are released: https://sid2697.github.io/epic-contact.