Scaffold Diffusion: Sparse Multi-Category Voxel Structure Generation with Discrete Diffusion
This addresses the difficulty of generating sparse 3D voxel structures for applications like Minecraft house generation, though it is incremental as it adapts existing discrete diffusion methods to a new domain.
The paper tackles the problem of generating realistic sparse multi-category 3D voxel structures, which is challenging due to cubic memory scaling and class imbalance from sparsity, and introduces Scaffold Diffusion, a generative model that uses discrete diffusion to produce coherent structures even with over 98% sparsity in training data.
Generating realistic sparse multi-category 3D voxel structures is difficult due to the cubic memory scaling of voxel structures and moreover the significant class imbalance caused by sparsity. We introduce Scaffold Diffusion, a generative model designed for sparse multi-category 3D voxel structures. By treating voxels as tokens, Scaffold Diffusion uses a discrete diffusion language model to generate 3D voxel structures. We show that discrete diffusion language models can be extended beyond inherently sequential domains such as text to generate spatially coherent 3D structures. We evaluate on Minecraft house structures from the 3D-Craft dataset and demonstrate that, unlike prior baselines and an auto-regressive formulation, Scaffold Diffusion produces realistic and coherent structures even when trained on data with over 98% sparsity. We provide an interactive viewer where readers can visualize generated samples and the generation process: https://scaffold.deepexploration.org/