$PC^2$: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstruction
This addresses the challenging problem of reconstructing 3D shapes from single RGB images for computer vision applications, offering a novel method that improves performance on real-world data.
The paper tackles single-image 3D reconstruction by generating sparse point clouds using a conditional denoising diffusion process with projection conditioning, achieving high-resolution geometries aligned with input images and enabling multiple shape generations from a single image, with large qualitative improvements on real-world data.
Reconstructing the 3D shape of an object from a single RGB image is a long-standing and highly challenging problem in computer vision. In this paper, we propose a novel method for single-image 3D reconstruction which generates a sparse point cloud via a conditional denoising diffusion process. Our method takes as input a single RGB image along with its camera pose and gradually denoises a set of 3D points, whose positions are initially sampled randomly from a three-dimensional Gaussian distribution, into the shape of an object. The key to our method is a geometrically-consistent conditioning process which we call projection conditioning: at each step in the diffusion process, we project local image features onto the partially-denoised point cloud from the given camera pose. This projection conditioning process enables us to generate high-resolution sparse geometries that are well-aligned with the input image, and can additionally be used to predict point colors after shape reconstruction. Moreover, due to the probabilistic nature of the diffusion process, our method is naturally capable of generating multiple different shapes consistent with a single input image. In contrast to prior work, our approach not only performs well on synthetic benchmarks, but also gives large qualitative improvements on complex real-world data.