C3G: Learning Compact 3D Representations with 2K Gaussians
This work addresses the challenge of high memory usage and sub-optimal feature aggregation in 3D scene reconstruction and understanding for computer vision applications, representing an incremental improvement over prior methods.
The paper tackles the problem of reconstructing and understanding 3D scenes from sparse views by proposing C3G, a framework that estimates compact 3D Gaussians at essential locations, which reduces memory overhead and improves feature aggregation, achieving superior memory efficiency and feature fidelity compared to existing methods.
Reconstructing and understanding 3D scenes from unposed sparse views in a feed-forward manner remains as a challenging task in 3D computer vision. Recent approaches use per-pixel 3D Gaussian Splatting for reconstruction, followed by a 2D-to-3D feature lifting stage for scene understanding. However, they generate excessive redundant Gaussians, causing high memory overhead and sub-optimal multi-view feature aggregation, leading to degraded novel view synthesis and scene understanding performance. We propose C3G, a novel feed-forward framework that estimates compact 3D Gaussians only at essential spatial locations, minimizing redundancy while enabling effective feature lifting. We introduce learnable tokens that aggregate multi-view features through self-attention to guide Gaussian generation, ensuring each Gaussian integrates relevant visual features across views. We then exploit the learned attention patterns for Gaussian decoding to efficiently lift features. Extensive experiments on pose-free novel view synthesis, 3D open-vocabulary segmentation, and view-invariant feature aggregation demonstrate our approach's effectiveness. Results show that a compact yet geometrically meaningful representation is sufficient for high-quality scene reconstruction and understanding, achieving superior memory efficiency and feature fidelity compared to existing methods.