A cost-effective method for improving and re-purposing large, pre-trained GANs by fine-tuning their class-embeddings
This provides a cost-effective solution for researchers and engineers to modify and repurpose large generative models, addressing computational and bias issues, though it is incremental as it builds on existing BigGANs.
The paper tackles the problem of fine-tuning large pre-trained GANs like BigGANs, which is computationally expensive and unstable, by proposing a method that fine-tunes only the class-embedding layer. The result shows effectiveness in improving realism and diversity for mode-collapse classes, repurposing for new datasets, and de-biasing, with significant gains in sample quality and diversity.
Large, pre-trained generative models have been increasingly popular and useful to both the research and wider communities. Specifically, BigGANs a class-conditional Generative Adversarial Networks trained on ImageNet---achieved excellent, state-of-the-art capability in generating realistic photos. However, fine-tuning or training BigGANs from scratch is practically impossible for most researchers and engineers because (1) GAN training is often unstable and suffering from mode-collapse; and (2) the training requires a significant amount of computation, 256 Google TPUs for 2 days or 8xV100 GPUs for 15 days. Importantly, many pre-trained generative models both in NLP and image domains were found to contain biases that are harmful to society. Thus, we need computationally-feasible methods for modifying and re-purposing these huge, pre-trained models for downstream tasks. In this paper, we propose a cost-effective optimization method for improving and re-purposing BigGANs by fine-tuning only the class-embedding layer. We show the effectiveness of our model-editing approach in three tasks: (1) significantly improving the realism and diversity of samples of complete mode-collapse classes; (2) re-purposing ImageNet BigGANs for generating images for Places365; and (3) de-biasing or improving the sample diversity for selected ImageNet classes.