NVGS: Neural Visibility for Occlusion Culling in 3D Gaussian Splatting
This addresses a bottleneck in rendering efficiency for 3D scenes, offering incremental improvements for applications like real-time graphics and virtual reality.
The paper tackles the problem of occlusion culling in 3D Gaussian Splatting, which is hindered by the semi-transparent nature of Gaussians, by proposing a method that learns a viewpoint-dependent visibility function using a shared MLP to discard occluded primitives during rendering, resulting in improved VRAM usage and image quality compared to the state of the art.
3D Gaussian Splatting can exploit frustum culling and level-of-detail strategies to accelerate rendering of scenes containing a large number of primitives. However, the semi-transparent nature of Gaussians prevents the application of another highly effective technique: occlusion culling. We address this limitation by proposing a novel method to learn the viewpoint-dependent visibility function of all Gaussians in a trained model using a small, shared MLP across instances of an asset in a scene. By querying it for Gaussians within the viewing frustum prior to rasterization, our method can discard occluded primitives during rendering. Leveraging Tensor Cores for efficient computation, we integrate these neural queries directly into a novel instanced software rasterizer. Our approach outperforms the current state of the art for composed scenes in terms of VRAM usage and image quality, utilizing a combination of our instanced rasterizer and occlusion culling MLP, and exhibits complementary properties to existing LoD techniques.