NeuralPassthrough: Learned Real-Time View Synthesis for VR
This addresses the limitation of VR headsets blocking users from seeing their physical environment, offering a solution for immersive VR applications, though it appears incremental as it builds on existing passthrough techniques with a novel learning approach.
The paper tackles the problem of real-time view synthesis for VR passthrough, which suffers from artifacts and poor image quality due to inaccurate depth and limited camera capabilities, by proposing a learned method that demonstrates superior image quality compared to state-of-the-art techniques while meeting real-time VR requirements.
Virtual reality (VR) headsets provide an immersive, stereoscopic visual experience, but at the cost of blocking users from directly observing their physical environment. Passthrough techniques are intended to address this limitation by leveraging outward-facing cameras to reconstruct the images that would otherwise be seen by the user without the headset. This is inherently a real-time view synthesis challenge, since passthrough cameras cannot be physically co-located with the eyes. Existing passthrough techniques suffer from distracting reconstruction artifacts, largely due to the lack of accurate depth information (especially for near-field and disoccluded objects), and also exhibit limited image quality (e.g., being low resolution and monochromatic). In this paper, we propose the first learned passthrough method and assess its performance using a custom VR headset that contains a stereo pair of RGB cameras. Through both simulations and experiments, we demonstrate that our learned passthrough method delivers superior image quality compared to state-of-the-art methods, while meeting strict VR requirements for real-time, perspective-correct stereoscopic view synthesis over a wide field of view for desktop-connected headsets.