clip2latent: Text driven sampling of a pre-trained StyleGAN using denoising diffusion and CLIP
This provides an efficient text-to-image generation method for users needing high-quality outputs without extensive data or compute, though it is incremental as it builds on existing CLIP and StyleGAN technologies.
The authors tackled the problem of creating text-to-image models without external data or fine-tuning by introducing clip2latent, which uses a diffusion model conditioned on CLIP embeddings to sample latent vectors from a pre-trained StyleGAN, enabling high-resolution (1024x1024 pixels) image generation with fast sampling and low training requirements.
We introduce a new method to efficiently create text-to-image models from a pre-trained CLIP and StyleGAN. It enables text driven sampling with an existing generative model without any external data or fine-tuning. This is achieved by training a diffusion model conditioned on CLIP embeddings to sample latent vectors of a pre-trained StyleGAN, which we call clip2latent. We leverage the alignment between CLIP's image and text embeddings to avoid the need for any text labelled data for training the conditional diffusion model. We demonstrate that clip2latent allows us to generate high-resolution (1024x1024 pixels) images based on text prompts with fast sampling, high image quality, and low training compute and data requirements. We also show that the use of the well studied StyleGAN architecture, without further fine-tuning, allows us to directly apply existing methods to control and modify the generated images adding a further layer of control to our text-to-image pipeline.