LiveNVS: Neural View Synthesis on Live RGB-D Streams
This addresses the need for real-time, high-quality visualization in live reconstruction for users like AR/VR developers, though it is incremental as it builds on existing neural rendering methods.
The paper tackles the problem of real-time photo-realistic visualization in RGB-D reconstruction by introducing LiveNVS, a system that enables neural novel view synthesis on live streams with low latency, achieving state-of-the-art rendering quality during capture.
Existing real-time RGB-D reconstruction approaches, like Kinect Fusion, lack real-time photo-realistic visualization. This is due to noisy, oversmoothed or incomplete geometry and blurry textures which are fused from imperfect depth maps and camera poses. Recent neural rendering methods can overcome many of such artifacts but are mostly optimized for offline usage, hindering the integration into a live reconstruction pipeline. In this paper, we present LiveNVS, a system that allows for neural novel view synthesis on a live RGB-D input stream with very low latency and real-time rendering. Based on the RGB-D input stream, novel views are rendered by projecting neural features into the target view via a densely fused depth map and aggregating the features in image-space to a target feature map. A generalizable neural network then translates the target feature map into a high-quality RGB image. LiveNVS achieves state-of-the-art neural rendering quality of unknown scenes during capturing, allowing users to virtually explore the scene and assess reconstruction quality in real-time.