One-Shot Adaptation of GAN in Just One CLIP
This addresses a specific challenge in image generation for researchers and practitioners, but it is incremental as it builds on existing GAN adaptation techniques.
The paper tackles the problem of overfitting or underfitting in single-shot GAN adaptation by proposing a method that uses CLIP space manipulations, resulting in diverse outputs with target texture and outperforming baselines both qualitatively and quantitatively.
There are many recent research efforts to fine-tune a pre-trained generator with a few target images to generate images of a novel domain. Unfortunately, these methods often suffer from overfitting or under-fitting when fine-tuned with a single target image. To address this, here we present a novel single-shot GAN adaptation method through unified CLIP space manipulations. Specifically, our model employs a two-step training strategy: reference image search in the source generator using a CLIP-guided latent optimization, followed by generator fine-tuning with a novel loss function that imposes CLIP space consistency between the source and adapted generators. To further improve the adapted model to produce spatially consistent samples with respect to the source generator, we also propose contrastive regularization for patchwise relationships in the CLIP space. Experimental results show that our model generates diverse outputs with the target texture and outperforms the baseline models both qualitatively and quantitatively. Furthermore, we show that our CLIP space manipulation strategy allows more effective attribute editing.