LT3SD: Latent Trees for 3D Scene Diffusion
This work addresses the challenge of scalable and detailed 3D scene generation for applications in computer graphics and AI, representing an incremental improvement over existing methods.
The paper tackles the problem of generating large-scale 3D scenes, which previous diffusion models struggled with in terms of spatial extent and quality, by introducing a latent tree representation and a diffusion process that enables high-quality, unconditional generation and probabilistic completion of scenes.
We present LT3SD, a novel latent diffusion model for large-scale 3D scene generation. Recent advances in diffusion models have shown impressive results in 3D object generation, but are limited in spatial extent and quality when extended to 3D scenes. To generate complex and diverse 3D scene structures, we introduce a latent tree representation to effectively encode both lower-frequency geometry and higher-frequency detail in a coarse-to-fine hierarchy. We can then learn a generative diffusion process in this latent 3D scene space, modeling the latent components of a scene at each resolution level. To synthesize large-scale scenes with varying sizes, we train our diffusion model on scene patches and synthesize arbitrary-sized output 3D scenes through shared diffusion generation across multiple scene patches. Through extensive experiments, we demonstrate the efficacy and benefits of LT3SD for large-scale, high-quality unconditional 3D scene generation and for probabilistic completion for partial scene observations.