VEGS: View Extrapolation of Urban Scenes in 3D Gaussian Splatting using Learned Priors
This addresses a limitation in neural rendering for urban scenes, enabling better view synthesis beyond typical driving trajectories, though it appears incremental by building on existing 3D Gaussian Splatting techniques.
The paper tackles the Extrapolated View Synthesis (EVS) problem in urban scene reconstruction by improving rendering quality for views outside the training camera distribution, such as looking left, right, or downwards, using methods like dense LiDAR map initialization and learned priors, with qualitative and quantitative results demonstrating effectiveness.
Neural rendering-based urban scene reconstruction methods commonly rely on images collected from driving vehicles with cameras facing and moving forward. Although these methods can successfully synthesize from views similar to training camera trajectory, directing the novel view outside the training camera distribution does not guarantee on-par performance. In this paper, we tackle the Extrapolated View Synthesis (EVS) problem by evaluating the reconstructions on views such as looking left, right or downwards with respect to training camera distributions. To improve rendering quality for EVS, we initialize our model by constructing dense LiDAR map, and propose to leverage prior scene knowledge such as surface normal estimator and large-scale diffusion model. Qualitative and quantitative comparisons demonstrate the effectiveness of our methods on EVS. To the best of our knowledge, we are the first to address the EVS problem in urban scene reconstruction. Link to our project page: https://vegs3d.github.io/.