Score-Guided Diffusion for 3D Human Recovery
This addresses the problem of accurate 3D human recovery for computer vision applications, offering a novel method that is incremental in applying diffusion models to this domain.
The paper tackles 3D human pose and shape reconstruction from images by proposing ScoreHMR, which uses score guidance in a diffusion model's latent space to solve inverse problems without retraining, and it outperforms optimization baselines across multiple benchmarks.
We present Score-Guided Human Mesh Recovery (ScoreHMR), an approach for solving inverse problems for 3D human pose and shape reconstruction. These inverse problems involve fitting a human body model to image observations, traditionally solved through optimization techniques. ScoreHMR mimics model fitting approaches, but alignment with the image observation is achieved through score guidance in the latent space of a diffusion model. The diffusion model is trained to capture the conditional distribution of the human model parameters given an input image. By guiding its denoising process with a task-specific score, ScoreHMR effectively solves inverse problems for various applications without the need for retraining the task-agnostic diffusion model. We evaluate our approach on three settings/applications. These are: (i) single-frame model fitting; (ii) reconstruction from multiple uncalibrated views; (iii) reconstructing humans in video sequences. ScoreHMR consistently outperforms all optimization baselines on popular benchmarks across all settings. We make our code and models available at the https://statho.github.io/ScoreHMR.