Global Latent Neural Rendering
This work addresses the challenge of synthesizing novel views from sparse inputs for applications in computer vision and graphics, representing an incremental improvement over prior methods.
The paper tackles the problem of generalizable novel view synthesis by proposing a global rendering operator that processes all camera rays jointly, instead of treating pixels independently, and demonstrates that their method consistently outperforms existing approaches by significant margins.
A recent trend among generalizable novel view synthesis methods is to learn a rendering operator acting over single camera rays. This approach is promising because it removes the need for explicit volumetric rendering, but it effectively treats target images as collections of independent pixels. Here, we propose to learn a global rendering operator acting over all camera rays jointly. We show that the right representation to enable such rendering is a 5-dimensional plane sweep volume consisting of the projection of the input images on a set of planes facing the target camera. Based on this understanding, we introduce our Convolutional Global Latent Renderer (ConvGLR), an efficient convolutional architecture that performs the rendering operation globally in a low-resolution latent space. Experiments on various datasets under sparse and generalizable setups show that our approach consistently outperforms existing methods by significant margins.