CompNVS: Novel View Synthesis with Scene Completion
This addresses the problem of generating realistic 3D scenes from limited data for applications like virtual reality or robotics, representing an incremental advance in combining scene completion with view synthesis.
The paper tackles novel view synthesis from incomplete RGB-D images by proposing a generative pipeline that completes unobserved scene parts, achieving photorealistic results that outperform state-of-the-art methods, particularly in unobserved areas.
We introduce a scalable framework for novel view synthesis from RGB-D images with largely incomplete scene coverage. While generative neural approaches have demonstrated spectacular results on 2D images, they have not yet achieved similar photorealistic results in combination with scene completion where a spatial 3D scene understanding is essential. To this end, we propose a generative pipeline performing on a sparse grid-based neural scene representation to complete unobserved scene parts via a learned distribution of scenes in a 2.5D-3D-2.5D manner. We process encoded image features in 3D space with a geometry completion network and a subsequent texture inpainting network to extrapolate the missing area. Photorealistic image sequences can be finally obtained via consistency-relevant differentiable rendering. Comprehensive experiments show that the graphical outputs of our method outperform the state of the art, especially within unobserved scene parts.