Native and Compact Structured Latents for 3D Generation
This addresses a key bottleneck in 3D generative modeling for applications like gaming and simulation, representing a significant advancement rather than an incremental improvement.
The paper tackles the challenge of generating realistic 3D assets with complex topologies and detailed appearance by introducing O-Voxel, a structured latent representation that encodes geometry and appearance, and a Sparse Compression VAE, resulting in generated assets that far exceed existing models in geometry and material quality.
Recent advancements in 3D generative modeling have significantly improved the generation realism, yet the field is still hampered by existing representations, which struggle to capture assets with complex topologies and detailed appearance. This paper present an approach for learning a structured latent representation from native 3D data to address this challenge. At its core is a new sparse voxel structure called O-Voxel, an omni-voxel representation that encodes both geometry and appearance. O-Voxel can robustly model arbitrary topology, including open, non-manifold, and fully-enclosed surfaces, while capturing comprehensive surface attributes beyond texture color, such as physically-based rendering parameters. Based on O-Voxel, we design a Sparse Compression VAE which provides a high spatial compression rate and a compact latent space. We train large-scale flow-matching models comprising 4B parameters for 3D generation using diverse public 3D asset datasets. Despite their scale, inference remains highly efficient. Meanwhile, the geometry and material quality of our generated assets far exceed those of existing models. We believe our approach offers a significant advancement in 3D generative modeling.