DIP-GS: Deep Image Prior For Gaussian Splatting Sparse View Recovery
This addresses a bottleneck in 3D scene reconstruction for applications requiring sparse inputs, but it is incremental as it builds on existing 3DGS and DIP techniques.
The paper tackles the problem of sparse view reconstruction in 3D Gaussian Splatting (3DGS), which struggles with limited input views, by proposing DIP-GS, a method that integrates Deep Image Prior (DIP) to enable recovery in such scenarios without pre-trained models, achieving state-of-the-art competitive results.
3D Gaussian Splatting (3DGS) is a leading 3D scene reconstruction method, obtaining high-quality reconstruction with real-time rendering runtime performance. The main idea behind 3DGS is to represent the scene as a collection of 3D gaussians, while learning their parameters to fit the given views of the scene. While achieving superior performance in the presence of many views, 3DGS struggles with sparse view reconstruction, where the input views are sparse and do not fully cover the scene and have low overlaps. In this paper, we propose DIP-GS, a Deep Image Prior (DIP) 3DGS representation. By using the DIP prior, which utilizes internal structure and patterns, with coarse-to-fine manner, DIP-based 3DGS can operate in scenarios where vanilla 3DGS fails, such as sparse view recovery. Note that our approach does not use any pre-trained models such as generative models and depth estimation, but rather relies only on the input frames. Among such methods, DIP-GS obtains state-of-the-art (SOTA) competitive results on various sparse-view reconstruction tasks, demonstrating its capabilities.