DiffGS: Functional Gaussian Splatting Diffusion
This work addresses a specific problem in 3D graphics and AI for researchers and practitioners by introducing a novel method to generate Gaussian Splatting, though it appears incremental as it builds upon existing 3DGS and diffusion models.
The paper tackles the challenge of generating 3D Gaussian Splatting (3DGS) due to its discreteness and unstructured nature by proposing DiffGS, a latent diffusion model that generates Gaussian primitives at arbitrary numbers for high-fidelity rendering, achieving efficient 3D generation across tasks like unconditional and conditional generation from text, image, and partial 3DGS.
3D Gaussian Splatting (3DGS) has shown convincing performance in rendering speed and fidelity, yet the generation of Gaussian Splatting remains a challenge due to its discreteness and unstructured nature. In this work, we propose DiffGS, a general Gaussian generator based on latent diffusion models. DiffGS is a powerful and efficient 3D generative model which is capable of generating Gaussian primitives at arbitrary numbers for high-fidelity rendering with rasterization. The key insight is to represent Gaussian Splatting in a disentangled manner via three novel functions to model Gaussian probabilities, colors and transforms. Through the novel disentanglement of 3DGS, we represent the discrete and unstructured 3DGS with continuous Gaussian Splatting functions, where we then train a latent diffusion model with the target of generating these Gaussian Splatting functions both unconditionally and conditionally. Meanwhile, we introduce a discretization algorithm to extract Gaussians at arbitrary numbers from the generated functions via octree-guided sampling and optimization. We explore DiffGS for various tasks, including unconditional generation, conditional generation from text, image, and partial 3DGS, as well as Point-to-Gaussian generation. We believe that DiffGS provides a new direction for flexibly modeling and generating Gaussian Splatting.