Off The Grid: Detection of Primitives for Feed-Forward 3D Gaussian Splatting
This work addresses a bottleneck in real-time 3D scene generation for applications like novel view synthesis, offering improved efficiency and quality.
The paper tackles the problem of suboptimal primitive placement in feed-forward 3D Gaussian Splatting models, which limits quality and efficiency, by introducing an adaptive detection method that achieves state-of-the-art novel view synthesis with fewer primitives and reduces artifacts.
Feed-forward 3D Gaussian Splatting (3DGS) models enable real-time scene generation but are hindered by suboptimal pixel-aligned primitive placement, which relies on a dense, rigid grid and limits both quality and efficiency. We introduce a new feed-forward architecture that detects 3D Gaussian primitives at a sub-pixel level, replacing the pixel grid with an adaptive, "Off The Grid" distribution. Inspired by keypoint detection, our multi-resolution decoder learns to distribute primitives across image patches. This module is trained end-to-end with a 3D reconstruction backbone using self-supervised learning. Our resulting pose-free model generates photorealistic scenes in seconds, achieving state-of-the-art novel view synthesis for feed-forward models. It outperforms competitors while using far fewer primitives, demonstrating a more accurate and efficient allocation that captures fine details and reduces artifacts. Moreover, we observe that by learning to render 3D Gaussians, our 3D reconstruction backbone improves camera pose estimation, suggesting opportunities to train these foundational models without labels.