3D-aware Image Generation and Editing with Multi-modal Conditions
This addresses the challenge of poor disentanglement and lack of multi-modal control in 3D-aware image generation for computer graphics and vision applications, representing an incremental advancement.
The paper tackles the problem of 3D-consistent image generation from 2D semantic labels by proposing a model that improves disentanglement of shape and appearance and enables multi-modal control with noise, text, and reference images, achieving superior qualitative and quantitative results compared to existing methods.
3D-consistent image generation from a single 2D semantic label is an important and challenging research topic in computer graphics and computer vision. Although some related works have made great progress in this field, most of the existing methods suffer from poor disentanglement performance of shape and appearance, and lack multi-modal control. In this paper, we propose a novel end-to-end 3D-aware image generation and editing model incorporating multiple types of conditional inputs, including pure noise, text and reference image. On the one hand, we dive into the latent space of 3D Generative Adversarial Networks (GANs) and propose a novel disentanglement strategy to separate appearance features from shape features during the generation process. On the other hand, we propose a unified framework for flexible image generation and editing tasks with multi-modal conditions. Our method can generate diverse images with distinct noises, edit the attribute through a text description and conduct style transfer by giving a reference RGB image. Extensive experiments demonstrate that the proposed method outperforms alternative approaches both qualitatively and quantitatively on image generation and editing.