SSPU-Net: Self-Supervised Point Cloud Upsampling via Differentiable Rendering
This addresses the need for dense point cloud generation in 3D vision applications, offering a self-supervised alternative to supervised methods, though it is incremental as it builds on existing upsampling and rendering techniques.
The paper tackles the problem of upsampling sparse 3D point clouds without dense ground truth by proposing a self-supervised method that uses differentiable rendering and consistency losses, achieving impressive results on CAD and scanned datasets.
Point clouds obtained from 3D sensors are usually sparse. Existing methods mainly focus on upsampling sparse point clouds in a supervised manner by using dense ground truth point clouds. In this paper, we propose a self-supervised point cloud upsampling network (SSPU-Net) to generate dense point clouds without using ground truth. To achieve this, we exploit the consistency between the input sparse point cloud and generated dense point cloud for the shapes and rendered images. Specifically, we first propose a neighbor expansion unit (NEU) to upsample the sparse point clouds, where the local geometric structures of the sparse point clouds are exploited to learn weights for point interpolation. Then, we develop a differentiable point cloud rendering unit (DRU) as an end-to-end module in our network to render the point cloud into multi-view images. Finally, we formulate a shape-consistent loss and an image-consistent loss to train the network so that the shapes of the sparse and dense point clouds are as consistent as possible. Extensive results on the CAD and scanned datasets demonstrate that our method can achieve impressive results in a self-supervised manner. Code is available at https://github.com/fpthink/SSPU-Net.