XDGAN: Multi-Modal 3D Shape Generation in 2D Space
This addresses the challenge of slow and inflexible 3D generative models for applications in computer graphics and AI, offering a novel approach but is incremental in adapting existing 2D methods to 3D.
The paper tackles the problem of generating 3D shapes by leveraging efficient 2D image generative models, proposing XDGAN to convert 3D shapes into geometry images for fast synthesis and editing, showing effectiveness in tasks like 3D shape generation and single-view reconstruction with significant speed and flexibility improvements.
Generative models for 2D images has recently seen tremendous progress in quality, resolution and speed as a result of the efficiency of 2D convolutional architectures. However it is difficult to extend this progress into the 3D domain since most current 3D representations rely on custom network components. This paper addresses a central question: Is it possible to directly leverage 2D image generative models to generate 3D shapes instead? To answer this, we propose XDGAN, an effective and fast method for applying 2D image GAN architectures to the generation of 3D object geometry combined with additional surface attributes, like color textures and normals. Specifically, we propose a novel method to convert 3D shapes into compact 1-channel geometry images and leverage StyleGAN3 and image-to-image translation networks to generate 3D objects in 2D space. The generated geometry images are quick to convert to 3D meshes, enabling real-time 3D object synthesis, visualization and interactive editing. Moreover, the use of standard 2D architectures can help bring more 2D advances into the 3D realm. We show both quantitatively and qualitatively that our method is highly effective at various tasks such as 3D shape generation, single view reconstruction and shape manipulation, while being significantly faster and more flexible compared to recent 3D generative models.