Follow My Hold: Hand-Object Interaction Reconstruction through Geometric Guidance
This addresses the challenge of high-quality 3D object reconstruction from single images for applications in robotics and AR/VR, though it is incremental as it builds on diffusion-based methods with novel guidance.
The paper tackles the problem of reconstructing 3D geometry of hand-held objects from monocular RGB images by using hand-object interaction as geometric guidance, resulting in accurate, robust, and coherent reconstructions under occlusion with generalization to in-the-wild scenarios.
We propose a novel diffusion-based framework for reconstructing 3D geometry of hand-held objects from monocular RGB images by leveraging hand-object interaction as geometric guidance. Our method conditions a latent diffusion model on an inpainted object appearance and uses inference-time guidance to optimize the object reconstruction, while simultaneously ensuring plausible hand-object interactions. Unlike prior methods that rely on extensive post-processing or produce low-quality reconstructions, our approach directly generates high-quality object geometry during the diffusion process by introducing guidance with an optimization-in-the-loop design. Specifically, we guide the diffusion model by applying supervision to the velocity field while simultaneously optimizing the transformations of both the hand and the object being reconstructed. This optimization is driven by multi-modal geometric cues, including normal and depth alignment, silhouette consistency, and 2D keypoint reprojection. We further incorporate signed distance field supervision and enforce contact and non-intersection constraints to ensure physical plausibility of hand-object interaction. Our method yields accurate, robust and coherent reconstructions under occlusion while generalizing well to in-the-wild scenarios.