Efficient Transfer Learning in Diffusion Models via Adversarial Noise
This addresses the data scarcity issue for researchers and practitioners using DPMs, offering an incremental improvement over prior transfer learning approaches.
The paper tackles the limited data problem in Diffusion Probabilistic Models (DPMs) for image generation by proposing TAN, a transfer learning method that improves efficiency and achieves superior image quality and diversity in few-shot tasks compared to existing methods.
Diffusion Probabilistic Models (DPMs) have demonstrated substantial promise in image generation tasks but heavily rely on the availability of large amounts of training data. Previous works, like GANs, have tackled the limited data problem by transferring pre-trained models learned with sufficient data. However, those methods are hard to be utilized in DPMs since the distinct differences between DPM-based and GAN-based methods, showing in the unique iterative denoising process integral and the need for many timesteps with no-targeted noise in DPMs. In this paper, we propose a novel DPMs-based transfer learning method, TAN, to address the limited data problem. It includes two strategies: similarity-guided training, which boosts transfer with a classifier, and adversarial noise selection which adaptive chooses targeted noise based on the input image. Extensive experiments in the context of few-shot image generation tasks demonstrate that our method is not only efficient but also excels in terms of image quality and diversity when compared to existing GAN-based and DDPM-based methods.