RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion
This addresses the challenge of 3D scene generation for applications in graphics and AI, offering a novel approach that does not require video or multi-view data, though it builds incrementally on diffusion-based techniques.
The paper tackles the problem of generating forward-facing 3D scenes from text descriptions by optimizing a 3D Gaussian Splatting representation using pretrained diffusion models, achieving high-quality results as shown by user study preferences of 88-95% over existing methods.
We introduce RealmDreamer, a technique for generating forward-facing 3D scenes from text descriptions. Our method optimizes a 3D Gaussian Splatting representation to match complex text prompts using pretrained diffusion models. Our key insight is to leverage 2D inpainting diffusion models conditioned on an initial scene estimate to provide low variance supervision for unknown regions during 3D distillation. In conjunction, we imbue high-fidelity geometry with geometric distillation from a depth diffusion model, conditioned on samples from the inpainting model. We find that the initialization of the optimization is crucial, and provide a principled methodology for doing so. Notably, our technique doesn't require video or multi-view data and can synthesize various high-quality 3D scenes in different styles with complex layouts. Further, the generality of our method allows 3D synthesis from a single image. As measured by a comprehensive user study, our method outperforms all existing approaches, preferred by 88-95%. Project Page: https://realmdreamer.github.io/