Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering
This work addresses a bottleneck in 3D scene rendering for applications requiring view-adaptive and efficient models, representing an incremental improvement over existing methods.
The paper tackles the problem of redundant Gaussians in 3D Gaussian Splatting for neural rendering, which reduces robustness to view changes and texture-less areas, by introducing Scaffold-GS, a method that uses anchor points to distribute local Gaussians and predicts attributes on-the-fly, resulting in effective reduction of redundant Gaussians while maintaining high-quality rendering and speed.
Neural rendering methods have significantly advanced photo-realistic 3D scene rendering in various academic and industrial applications. The recent 3D Gaussian Splatting method has achieved the state-of-the-art rendering quality and speed combining the benefits of both primitive-based representations and volumetric representations. However, it often leads to heavily redundant Gaussians that try to fit every training view, neglecting the underlying scene geometry. Consequently, the resulting model becomes less robust to significant view changes, texture-less area and lighting effects. We introduce Scaffold-GS, which uses anchor points to distribute local 3D Gaussians, and predicts their attributes on-the-fly based on viewing direction and distance within the view frustum. Anchor growing and pruning strategies are developed based on the importance of neural Gaussians to reliably improve the scene coverage. We show that our method effectively reduces redundant Gaussians while delivering high-quality rendering. We also demonstrates an enhanced capability to accommodate scenes with varying levels-of-detail and view-dependent observations, without sacrificing the rendering speed.