OGGSplat: Open Gaussian Growing for Generalizable Reconstruction with Expanded Field-of-View
This addresses the challenge of generalizable 3D reconstruction for applications like virtual reality and embodied AI, offering an incremental improvement over existing methods that struggle with regions outside the input view cone.
The paper tackles the problem of reconstructing semantic-aware 3D scenes from sparse views by proposing OGGSplat, which expands the field-of-view and achieves generalizable reconstruction with semantic consistency and visual plausibility, as demonstrated on a new Gaussian Outpainting benchmark and with smartphone-captured images.
Reconstructing semantic-aware 3D scenes from sparse views is a challenging yet essential research direction, driven by the demands of emerging applications such as virtual reality and embodied AI. Existing per-scene optimization methods require dense input views and incur high computational costs, while generalizable approaches often struggle to reconstruct regions outside the input view cone. In this paper, we propose OGGSplat, an open Gaussian growing method that expands the field-of-view in generalizable 3D reconstruction. Our key insight is that the semantic attributes of open Gaussians provide strong priors for image extrapolation, enabling both semantic consistency and visual plausibility. Specifically, once open Gaussians are initialized from sparse views, we introduce an RGB-semantic consistent inpainting module applied to selected rendered views. This module enforces bidirectional control between an image diffusion model and a semantic diffusion model. The inpainted regions are then lifted back into 3D space for efficient and progressive Gaussian parameter optimization. To evaluate our method, we establish a Gaussian Outpainting (GO) benchmark that assesses both semantic and generative quality of reconstructed open-vocabulary scenes. OGGSplat also demonstrates promising semantic-aware scene reconstruction capabilities when provided with two view images captured directly from a smartphone camera.