StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators
This enables domain adaptation for image generation without image data, benefiting researchers and practitioners in computer vision and generative AI, though it is incremental as it builds on CLIP and StyleGAN.
The paper tackles the problem of adapting image generators to new domains using only text prompts, without collecting any images, by leveraging CLIP models, and demonstrates that this method can achieve diverse style and shape modifications in minutes of training, which are difficult with existing methods.
Can a generative model be trained to produce images from a specific domain, guided by a text prompt only, without seeing any image? In other words: can an image generator be trained "blindly"? Leveraging the semantic power of large scale Contrastive-Language-Image-Pre-training (CLIP) models, we present a text-driven method that allows shifting a generative model to new domains, without having to collect even a single image. We show that through natural language prompts and a few minutes of training, our method can adapt a generator across a multitude of domains characterized by diverse styles and shapes. Notably, many of these modifications would be difficult or outright impossible to reach with existing methods. We conduct an extensive set of experiments and comparisons across a wide range of domains. These demonstrate the effectiveness of our approach and show that our shifted models maintain the latent-space properties that make generative models appealing for downstream tasks.