DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans
This addresses the challenge of generating detailed and diverse 3D human models from limited 2D data, which is important for applications like virtual reality and animation, though it builds incrementally on existing diffusion models.
The authors tackled the problem of photorealistic 3D human reconstruction from a single RGB image by proposing DiffHuman, a probabilistic method that predicts a distribution over 3D reconstructions, enabling sampling of multiple detailed avatars and achieving a 55x speed-up in runtime.
We present DiffHuman, a probabilistic method for photorealistic 3D human reconstruction from a single RGB image. Despite the ill-posed nature of this problem, most methods are deterministic and output a single solution, often resulting in a lack of geometric detail and blurriness in unseen or uncertain regions. In contrast, DiffHuman predicts a probability distribution over 3D reconstructions conditioned on an input 2D image, which allows us to sample multiple detailed 3D avatars that are consistent with the image. DiffHuman is implemented as a conditional diffusion model that denoises pixel-aligned 2D observations of an underlying 3D shape representation. During inference, we may sample 3D avatars by iteratively denoising 2D renders of the predicted 3D representation. Furthermore, we introduce a generator neural network that approximates rendering with considerably reduced runtime (55x speed up), resulting in a novel dual-branch diffusion framework. Our experiments show that DiffHuman can produce diverse and detailed reconstructions for the parts of the person that are unseen or uncertain in the input image, while remaining competitive with the state-of-the-art when reconstructing visible surfaces.