Cycle-Consistent Inverse GAN for Text-to-Image Synthesis
This addresses the problem of generating and manipulating images from text descriptions for applications in creative and automated content creation, representing an incremental improvement by unifying separate tasks into one framework.
The paper tackles text-to-image synthesis and manipulation by proposing a unified framework that trains a GAN without text, learns a cycle-consistent inversion model, and optimizes latent codes based on text similarity, achieving validated results on Recipe1M and CUB datasets.
This paper investigates an open research task of text-to-image synthesis for automatically generating or manipulating images from text descriptions. Prevailing methods mainly use the text as conditions for GAN generation, and train different models for the text-guided image generation and manipulation tasks. In this paper, we propose a novel unified framework of Cycle-consistent Inverse GAN (CI-GAN) for both text-to-image generation and text-guided image manipulation tasks. Specifically, we first train a GAN model without text input, aiming to generate images with high diversity and quality. Then we learn a GAN inversion model to convert the images back to the GAN latent space and obtain the inverted latent codes for each image, where we introduce the cycle-consistency training to learn more robust and consistent inverted latent codes. We further uncover the latent space semantics of the trained GAN model, by learning a similarity model between text representations and the latent codes. In the text-guided optimization module, we generate images with the desired semantic attributes by optimizing the inverted latent codes. Extensive experiments on the Recipe1M and CUB datasets validate the efficacy of our proposed framework.