Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search
This work addresses the problem of zero-shot image-to-text and text-to-image generation for general users, offering a novel approach to cross-modal synthesis.
This paper introduces CLIP-GLaSS, a zero-shot framework that generates images from captions or captions from images. It achieves this by using a genetic algorithm to search the latent space of generative networks (BigGAN, StyleGAN2, GPT2) to find outputs whose CLIP embeddings are most similar to the input.
In this research work we present CLIP-GLaSS, a novel zero-shot framework to generate an image (or a caption) corresponding to a given caption (or image). CLIP-GLaSS is based on the CLIP neural network, which, given an image and a descriptive caption, provides similar embeddings. Differently, CLIP-GLaSS takes a caption (or an image) as an input, and generates the image (or the caption) whose CLIP embedding is the most similar to the input one. This optimal image (or caption) is produced via a generative network, after an exploration by a genetic algorithm. Promising results are shown, based on the experimentation of the image Generators BigGAN and StyleGAN2, and of the text Generator GPT2