MVDD: Multi-View Depth Diffusion Models
This addresses the problem of 3D shape generation for applications like computer graphics and robotics, representing an incremental advance by adapting 2D diffusion methods to 3D data.
The paper tackles the challenge of generating 3D shapes by proposing MVDD, a diffusion model that uses multi-view depth to produce high-quality dense point clouds with 20K+ points and fine-grained details, achieving state-of-the-art results in 3D shape generation and depth completion.
Denoising diffusion models have demonstrated outstanding results in 2D image generation, yet it remains a challenge to replicate its success in 3D shape generation. In this paper, we propose leveraging multi-view depth, which represents complex 3D shapes in a 2D data format that is easy to denoise. We pair this representation with a diffusion model, MVDD, that is capable of generating high-quality dense point clouds with 20K+ points with fine-grained details. To enforce 3D consistency in multi-view depth, we introduce an epipolar line segment attention that conditions the denoising step for a view on its neighboring views. Additionally, a depth fusion module is incorporated into diffusion steps to further ensure the alignment of depth maps. When augmented with surface reconstruction, MVDD can also produce high-quality 3D meshes. Furthermore, MVDD stands out in other tasks such as depth completion, and can serve as a 3D prior, significantly boosting many downstream tasks, such as GAN inversion. State-of-the-art results from extensive experiments demonstrate MVDD's excellent ability in 3D shape generation, depth completion, and its potential as a 3D prior for downstream tasks.