Benchmarking Single-Step Inpainting Methods for Multi-Object 3D Gaussian Splatting Scenes
This work provides a benchmark and insights for researchers and practitioners working on 3D object removal and inpainting, particularly for 3D Gaussian Splatting scenes, by comparing different 2D inpainters and integration strategies.
This paper tackles the challenge of object removal and inpainting in 3D Gaussian Splatting (3DGS) scenes, focusing on 3D consistency. The authors found that reconstruction-based 2D inpainters outperform generative diffusion models in 3D consistency, and initializing a 3DGS scene from scratch yields higher quality inpainting results than finetuning existing scenes.
The tasks of object removal and inpainting 3D Gaussian Splatting (3DGS) scenes face challenges such as 3D consistency across camera views. In comparing 2D inpainters and their suitability for the 3D domain, we find that reconstruction-based inpainters outperform generative diffusion models in 3D consistency. Integrating these 2D inpainters into different single-step methods for creating and finetuning 3DGS scenes, our results indicate that initializing the scene from scratch produces higher quality results than finetuning the existing scene. Using a state-of-the-art generative 2D inpainter, we create a straightforward baseline to underline the importance of object removal before inpainting in the 3D setting. Since 360° datasets rarely include real-world ground truths, and challenging occlusion scenarios are equally sparse, we introduce a novel multi-object scene with recorded ground truth data and many views with object occlusions.