Diffusion-based 3D Hand Motion Recovery with Intuitive Physics
This work addresses the problem of 3D hand motion recovery for applications in computer vision and robotics, representing an incremental improvement by enhancing existing methods with physics integration.
The paper tackles the challenge of generating accurate and temporally coherent 3D hand motion from videos, especially during hand-object interactions, by introducing a diffusion-based and physics-augmented motion refinement model that improves frame-wise reconstruction methods and achieves state-of-the-art performance on benchmarks.
While 3D hand reconstruction from monocular images has made significant progress, generating accurate and temporally coherent motion estimates from videos remains challenging, particularly during hand-object interactions. In this paper, we present a novel 3D hand motion recovery framework that enhances image-based reconstructions through a diffusion-based and physics-augmented motion refinement model. Our model captures the distribution of refined motion estimates conditioned on initial ones, generating improved sequences through an iterative denoising process. Instead of relying on scarce annotated video data, we train our model only using motion capture data without images. We identify valuable intuitive physics knowledge during hand-object interactions, including key motion states and their associated motion constraints. We effectively integrate these physical insights into our diffusion model to improve its performance. Extensive experiments demonstrate that our approach significantly improves various frame-wise reconstruction methods, achieving state-of-the-art (SOTA) performance on existing benchmarks.