GaussFusion: Improving 3D Reconstruction in the Wild with A Geometry-Informed Video Generator
This addresses the problem of noisy and incomplete 3D reconstructions for applications like interactive 3D modeling, offering a robust solution that generalizes across different reconstruction methods.
The paper tackles artifacts in 3D Gaussian splatting reconstructions, such as floaters and flickering, by introducing a geometry-informed video generator that refines renderings, achieving state-of-the-art performance on novel-view synthesis benchmarks and enabling real-time operation at 21 FPS.
We present GaussFusion, a novel approach for improving 3D Gaussian splatting (3DGS) reconstructions in the wild through geometry-informed video generation. GaussFusion mitigates common 3DGS artifacts, including floaters, flickering, and blur caused by camera pose errors, incomplete coverage, and noisy geometry initialization. Unlike prior RGB-based approaches limited to a single reconstruction pipeline, our method introduces a geometry-informed video-to-video generator that refines 3DGS renderings across both optimization-based and feed-forward methods. Given an existing reconstruction, we render a Gaussian primitive video buffer encoding depth, normals, opacity, and covariance, which the generator refines to produce temporally coherent, artifact-free frames. We further introduce an artifact synthesis pipeline that simulates diverse degradation patterns, ensuring robustness and generalization. GaussFusion achieves state-of-the-art performance on novel-view synthesis benchmarks, and an efficient variant runs in real time at 21 FPS while maintaining similar performance, enabling interactive 3D applications.