Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization
This work addresses the challenge of expensive real data acquisition in transfer learning for domains with privacy and IP concerns, offering a novel method to enhance model performance with synthetic data, though it is incremental in improving existing transfer learning paradigms.
The paper tackled the problem of using synthetic images from text-to-image models for transfer learning, finding that naive incorporation fails due to distribution gaps, and introduced a two-stage bridged transfer framework and style inversion strategy, resulting in up to 30% accuracy improvements on classification tasks across multiple datasets and models.
Synthetic image data generation represents a promising avenue for training deep learning models, particularly in the realm of transfer learning, where obtaining real images within a specific domain can be prohibitively expensive due to privacy and intellectual property considerations. This work delves into the generation and utilization of synthetic images derived from text-to-image generative models in facilitating transfer learning paradigms. Despite the high visual fidelity of the generated images, we observe that their naive incorporation into existing real-image datasets does not consistently enhance model performance due to the inherent distribution gap between synthetic and real images. To address this issue, we introduce a novel two-stage framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability and subsequently uses real data for rapid adaptation. Alongside, We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements, with up to 30% accuracy increase on classification tasks. Intriguingly, we note that the enhancements were not yet saturated, indicating that the benefits may further increase with an expanded volume of synthetic data.