FWD: Real-time Novel View Synthesis with Forward Warping and Depth
This addresses the need for fast and high-quality view synthesis in applications like VR/AR, though it is incremental by building on existing depth-based and neural rendering techniques.
The paper tackles the problem of novel view synthesis from sparse inputs by proposing FWD, a method that achieves competitive quality to state-of-the-art approaches with a 130-1000x speedup and better perceptual quality, enabling real-time performance.
Novel view synthesis (NVS) is a challenging task requiring systems to generate photorealistic images of scenes from new viewpoints, where both quality and speed are important for applications. Previous image-based rendering (IBR) methods are fast, but have poor quality when input views are sparse. Recent Neural Radiance Fields (NeRF) and generalizable variants give impressive results but are not real-time. In our paper, we propose a generalizable NVS method with sparse inputs, called FWD, which gives high-quality synthesis in real-time. With explicit depth and differentiable rendering, it achieves competitive results to the SOTA methods with 130-1000x speedup and better perceptual quality. If available, we can seamlessly integrate sensor depth during either training or inference to improve image quality while retaining real-time speed. With the growing prevalence of depths sensors, we hope that methods making use of depth will become increasingly useful.