Text-to-Image Synthesis Based on Machine Generated Captions
This work addresses a data bottleneck in text-to-image synthesis for researchers and practitioners, but it is incremental as it builds on existing GAN and captioning methods.
The paper tackles the limited availability of captioned image datasets for text-to-image synthesis by proposing an approach that uses conditional GANs trained on uncaptioned images, with machine-generated captions from an Image Captioning Module, and reports preliminary but promising results compared to unconditional GANs on the LSUN bedroom dataset.
Text to Image Synthesis refers to the process of automatic generation of a photo-realistic image starting from a given text and is revolutionizing many real-world applications. In order to perform such process it is necessary to exploit datasets containing captioned images, meaning that each image is associated with one (or more) captions describing it. Despite the abundance of uncaptioned images datasets, the number of captioned datasets is limited. To address this issue, in this paper we propose an approach capable of generating images starting from a given text using conditional GANs trained on uncaptioned images dataset. In particular, uncaptioned images are fed to an Image Captioning Module to generate the descriptions. Then, the GAN Module is trained on both the input image and the machine-generated caption. To evaluate the results, the performance of our solution is compared with the results obtained by the unconditional GAN. For the experiments, we chose to use the uncaptioned dataset LSUN bedroom. The results obtained in our study are preliminary but still promising.