Neural Rays for Occlusion-aware Image-based Rendering
This addresses occlusion issues in image-based rendering for computer vision and graphics applications, representing an incremental improvement over existing generalization methods.
The paper tackles the problem of inconsistent image features from occluded views in neural radiance fields for novel view synthesis by introducing Neural Ray (NeuRay), which predicts visibility of 3D points to input views, resulting in state-of-the-art performance on generalization to unseen scenes and outperforming per-scene optimization methods after finetuning.
We present a new neural representation, called Neural Ray (NeuRay), for the novel view synthesis task. Recent works construct radiance fields from image features of input views to render novel view images, which enables the generalization to new scenes. However, due to occlusions, a 3D point may be invisible to some input views. On such a 3D point, these generalization methods will include inconsistent image features from invisible views, which interfere with the radiance field construction. To solve this problem, we predict the visibility of 3D points to input views within our NeuRay representation. This visibility enables the radiance field construction to focus on visible image features, which significantly improves its rendering quality. Meanwhile, a novel consistency loss is proposed to refine the visibility in NeuRay when finetuning on a specific scene. Experiments demonstrate that our approach achieves state-of-the-art performance on the novel view synthesis task when generalizing to unseen scenes and outperforms per-scene optimization methods after finetuning.