Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images
For 3D scene understanding from unposed images, COVScene provides a unified representation that jointly handles geometry, semantics, and occupancy without external calibration, addressing a key bottleneck in feed-forward Gaussian methods.
COVScene introduces a pose-free semantic Gaussian framework that couples renderable Gaussians with a dense semantic occupancy field via differentiable volumetric lifting, enabling comprehensive 3D scene understanding (novel view synthesis, open-vocabulary segmentation, occupancy prediction) from unposed images. On ScanNet and ScanNet++, it improves open-vocabulary segmentation and semantic occupancy prediction over self-supervised baselines while maintaining competitive rendering quality.
Comprehensive 3D scene understanding from sparse, unposed images requires a model to recover renderable geometry, open-vocabulary semantics, and free/occupied 3D space without relying on external camera calibration. Recent feed-forward Gaussian methods improve pose-free reconstruction and semantic rendering, but their Gaussian primitives are mainly optimized through image-space objectives and remain weakly constrained in unobserved regions. We propose \textit{COVScene}, a pose-free semantic Gaussian framework that couples renderable Gaussian primitives with a dense semantic occupancy field through differentiable volumetric lifting. Instead of converting Gaussians to voxels only at evaluation time, COVScene lifts the predicted semantic Gaussians inside the training computation graph, so volumetric regularization provides gradients to Gaussian opacity, geometry, and semantic features. The framework combines a semantic-aware Geometry Transformer, multi-task Gaussian decoding, geometric foundation distillation, and occupancy entropy regularization to support novel view synthesis, open-vocabulary semantic querying, and semantic occupancy prediction within a single representation. Experiments on ScanNet and ScanNet++ show that COVScene maintains competitive rendering quality, improves open-vocabulary segmentation, and achieves stronger semantic occupancy prediction than the self-supervised baseline without direct voxel-level supervision.