Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time
This work addresses a domain-specific problem in computer vision for applications like robotics and AR/VR, offering incremental improvements through semi-supervised learning.
The paper tackles the problem of estimating 3D hand and object poses from single images, which is challenging due to self-occlusion and scarce annotations, by proposing a semi-supervised learning framework that leverages spatial-temporal consistency in videos to generate pseudo labels, resulting in improved hand pose estimation and substantial gains in object pose estimation with better generalization across datasets.
Estimating 3D hand and object pose from a single image is an extremely challenging problem: hands and objects are often self-occluded during interactions, and the 3D annotations are scarce as even humans cannot directly label the ground-truths from a single image perfectly. To tackle these challenges, we propose a unified framework for estimating the 3D hand and object poses with semi-supervised learning. We build a joint learning framework where we perform explicit contextual reasoning between hand and object representations by a Transformer. Going beyond limited 3D annotations in a single image, we leverage the spatial-temporal consistency in large-scale hand-object videos as a constraint for generating pseudo labels in semi-supervised learning. Our method not only improves hand pose estimation in challenging real-world dataset, but also substantially improve the object pose which has fewer ground-truths per instance. By training with large-scale diverse videos, our model also generalizes better across multiple out-of-domain datasets. Project page and code: https://stevenlsw.github.io/Semi-Hand-Object