MonoSplat: Generalizable 3D Gaussian Splatting from Monocular Depth Foundation Models
This work addresses the problem of robust 3D reconstruction for real-time rendering applications, representing an incremental improvement by integrating pre-trained models into existing frameworks.
The paper tackles the challenge of limited generalizability in 3D Gaussian Splatting for novel scenes by introducing MonoSplat, which leverages monocular depth foundation models to achieve superior reconstruction quality and generalization, as demonstrated through extensive experiments on diverse datasets.
Recent advances in generalizable 3D Gaussian Splatting have demonstrated promising results in real-time high-fidelity rendering without per-scene optimization, yet existing approaches still struggle to handle unfamiliar visual content during inference on novel scenes due to limited generalizability. To address this challenge, we introduce MonoSplat, a novel framework that leverages rich visual priors from pre-trained monocular depth foundation models for robust Gaussian reconstruction. Our approach consists of two key components: a Mono-Multi Feature Adapter that transforms monocular features into multi-view representations, coupled with an Integrated Gaussian Prediction module that effectively fuses both feature types for precise Gaussian generation. Through the Adapter's lightweight attention mechanism, features are seamlessly aligned and aggregated across views while preserving valuable monocular priors, enabling the Prediction module to generate Gaussian primitives with accurate geometry and appearance. Through extensive experiments on diverse real-world datasets, we convincingly demonstrate that MonoSplat achieves superior reconstruction quality and generalization capability compared to existing methods while maintaining computational efficiency with minimal trainable parameters. Codes are available at https://github.com/CUHK-AIM-Group/MonoSplat.