PolyDiff: Generating 3D Polygonal Meshes with Diffusion Models
This addresses the need for realistic and diverse 3D mesh generation for downstream 3D workflows, representing a novel method rather than an incremental advancement.
PolyDiff tackles the problem of generating 3D polygonal meshes by introducing the first diffusion-based method that directly operates on mesh data, achieving significant improvements with an average FID and JSD improvement of 18.2 and 5.8 over state-of-the-art methods.
We introduce PolyDiff, the first diffusion-based approach capable of directly generating realistic and diverse 3D polygonal meshes. In contrast to methods that use alternate 3D shape representations (e.g. implicit representations), our approach is a discrete denoising diffusion probabilistic model that operates natively on the polygonal mesh data structure. This enables learning of both the geometric properties of vertices and the topological characteristics of faces. Specifically, we treat meshes as quantized triangle soups, progressively corrupted with categorical noise in the forward diffusion phase. In the reverse diffusion phase, a transformer-based denoising network is trained to revert the noising process, restoring the original mesh structure. At inference, new meshes can be generated by applying this denoising network iteratively, starting with a completely noisy triangle soup. Consequently, our model is capable of producing high-quality 3D polygonal meshes, ready for integration into downstream 3D workflows. Our extensive experimental analysis shows that PolyDiff achieves a significant advantage (avg. FID and JSD improvement of 18.2 and 5.8 respectively) over current state-of-the-art methods.