CLM: Removing the GPU Memory Barrier for 3D Gaussian Splatting
This work solves the GPU memory barrier for 3D Gaussian Splatting, enabling large-scale novel view synthesis on accessible hardware, though it is incremental as it optimizes an existing method.
The paper tackles the problem of scaling 3D Gaussian Splatting to large scenes by addressing its high GPU memory requirements, resulting in a system that enables rendering of scenes with 100 million Gaussians on a single consumer-grade GPU while maintaining state-of-the-art quality.
3D Gaussian Splatting (3DGS) is an increasingly popular novel view synthesis approach due to its fast rendering time, and high-quality output. However, scaling 3DGS to large (or intricate) scenes is challenging due to its large memory requirement, which exceed most GPU's memory capacity. In this paper, we describe CLM, a system that allows 3DGS to render large scenes using a single consumer-grade GPU, e.g., RTX4090. It does so by offloading Gaussians to CPU memory, and loading them into GPU memory only when necessary. To reduce performance and communication overheads, CLM uses a novel offloading strategy that exploits observations about 3DGS's memory access pattern for pipelining, and thus overlap GPU-to-CPU communication, GPU computation and CPU computation. Furthermore, we also exploit observation about the access pattern to reduce communication volume. Our evaluation shows that the resulting implementation can render a large scene that requires 100 million Gaussians on a single RTX4090 and achieve state-of-the-art reconstruction quality.