Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes
This work provides a foundation for efficient generative modeling of structured 3D environments, enabling interactive user workflows in Minecraft.
Dream-Cubed introduces a large-scale voxel dataset of Minecraft worlds and trains 3D diffusion models for controllable generation of interactive environments, achieving efficient inpainting/outpainting and demonstrating quality via human preference studies.
We introduce Dream-Cubed, a large-scale dataset of Minecraft worlds at voxel resolution, and a family of models using cubes as powerful compositional units for efficient generation of interactive 3D environments. Dream-Cubed comprises tens of billions of tokens from a carefully curated mixture of procedural biome terrain and high-quality human-authored maps. We use this dataset to conduct the first large-scale study of 3D diffusion models for voxel generation, analyzing discrete and continuous diffusion formulations, data compositions, and architectural design choices. Our models operate directly in the space of blocks, enabling efficient and semantically grounded generation while supporting interactive user workflows such as inpainting and outpainting from user-authored blocks. To quantitatively evaluate our models, we adapt the FID metric to assess semantic differences between real and generated world renderings, and validate generation quality through a human preference study. We release the full dataset, code, and all our pretrained models, which we hope will provide a foundation for future research in efficient generative modeling for structured, interactive 3D environments.