Point2Pix: Photo-Realistic Point Cloud Rendering via Neural Radiance Fields
This work addresses the problem of realistic image rendering from point clouds for applications in computer vision and graphics, representing an incremental improvement by integrating point cloud priors into NeRF pipelines.
The paper tackles the challenge of synthesizing photo-realistic images from sparse point clouds by introducing Point2Pix, a novel point renderer that links 3D point clouds with 2D pixels using Neural Radiance Fields, achieving high-quality image synthesis for novel indoor scenes as demonstrated on ScanNet and ArkitScenes datasets.
Synthesizing photo-realistic images from a point cloud is challenging because of the sparsity of point cloud representation. Recent Neural Radiance Fields and extensions are proposed to synthesize realistic images from 2D input. In this paper, we present Point2Pix as a novel point renderer to link the 3D sparse point clouds with 2D dense image pixels. Taking advantage of the point cloud 3D prior and NeRF rendering pipeline, our method can synthesize high-quality images from colored point clouds, generally for novel indoor scenes. To improve the efficiency of ray sampling, we propose point-guided sampling, which focuses on valid samples. Also, we present Point Encoding to build Multi-scale Radiance Fields that provide discriminative 3D point features. Finally, we propose Fusion Encoding to efficiently synthesize high-quality images. Extensive experiments on the ScanNet and ArkitScenes datasets demonstrate the effectiveness and generalization.