Diffusion Guided Domain Adaptation of Image Generators
This enables efficient domain adaptation of image generators for applications in creative and visual content generation, though it is incremental as it builds on existing diffusion and GAN techniques.
The paper tackles the problem of adapting GAN generators to new domains using text prompts without target domain samples, by leveraging a text-to-image diffusion model as a training objective. The result shows that their method achieves equally high CLIP scores and significantly lower FID than prior work on short prompts, and outperforms baselines on long and complicated prompts.
Can a text-to-image diffusion model be used as a training objective for adapting a GAN generator to another domain? In this paper, we show that the classifier-free guidance can be leveraged as a critic and enable generators to distill knowledge from large-scale text-to-image diffusion models. Generators can be efficiently shifted into new domains indicated by text prompts without access to groundtruth samples from target domains. We demonstrate the effectiveness and controllability of our method through extensive experiments. Although not trained to minimize CLIP loss, our model achieves equally high CLIP scores and significantly lower FID than prior work on short prompts, and outperforms the baseline qualitatively and quantitatively on long and complicated prompts. To our best knowledge, the proposed method is the first attempt at incorporating large-scale pre-trained diffusion models and distillation sampling for text-driven image generator domain adaptation and gives a quality previously beyond possible. Moreover, we extend our work to 3D-aware style-based generators and DreamBooth guidance.