Learning to Estimate Pose and Shape of Hand-Held Objects from RGB Images
This addresses the challenge of modeling hand-object interactions for robotics and AR/VR applications, but it is incremental as it builds on prior hand pose estimation and uses synthetic data with translation to real images.
The paper tackles the problem of estimating 3D pose and shape of hand-held objects from RGB images, showing that incorporating hand information improves object pose and shape estimation, with quantitative experiments indicating that even noisy hand parameters significantly help.
We develop a system for modeling hand-object interactions in 3D from RGB images that show a hand which is holding a novel object from a known category. We design a Convolutional Neural Network (CNN) for Hand-held Object Pose and Shape estimation called HOPS-Net and utilize prior work to estimate the hand pose and configuration. We leverage the insight that information about the hand facilitates object pose and shape estimation by incorporating the hand into both training and inference of the object pose and shape as well as the refinement of the estimated pose. The network is trained on a large synthetic dataset of objects in interaction with a human hand. To bridge the gap between real and synthetic images, we employ an image-to-image translation model (Augmented CycleGAN) that generates realistically textured objects given a synthetic rendering. This provides a scalable way of generating annotated data for training HOPS-Net. Our quantitative experiments show that even noisy hand parameters significantly help object pose and shape estimation. The qualitative experiments show results of pose and shape estimation of objects held by a hand "in the wild".