Enhancement of Novel View Synthesis Using Omnidirectional Image Completion
This work improves novel view synthesis for applications like virtual reality or 3D reconstruction, but it is incremental as it builds on existing NeRF-based approaches.
The study tackled the problem of synthesizing novel views from a single 360-degree RGB-D image by addressing artifacts from missing regions, using a method that completes these regions with a 2D image generative model and selects consistent images via simulated annealing, resulting in plausible novel views for artificial and real-world data.
In this study, we present a method for synthesizing novel views from a single 360-degree RGB-D image based on the neural radiance field (NeRF) . Prior studies relied on the neighborhood interpolation capability of multi-layer perceptrons to complete missing regions caused by occlusion and zooming, which leads to artifacts. In the method proposed in this study, the input image is reprojected to 360-degree RGB images at other camera positions, the missing regions of the reprojected images are completed by a 2D image generative model, and the completed images are utilized to train the NeRF. Because multiple completed images contain inconsistencies in 3D, we introduce a method to learn the NeRF model using a subset of completed images that cover the target scene with less overlap of completed regions. The selection of such a subset of images can be attributed to the maximum weight independent set problem, which is solved through simulated annealing. Experiments demonstrated that the proposed method can synthesize plausible novel views while preserving the features of the scene for both artificial and real-world data.