Generating Synthetic Data via Augmentations for Improved Facial Resemblance in DreamBooth and InstantID
This work addresses facial resemblance challenges in personalized text-to-image generation for professional portrait creation from amateur photos, representing an incremental improvement in augmentation strategies.
The paper tackles the problem of maintaining facial resemblance in personalized Stable Diffusion for portrait generation by evaluating augmentation strategies on DreamBooth and InstantID, finding that generative augmentation with InstantID improves fidelity when balanced with real images, as confirmed by a user study with 97 participants.
Personalizing Stable Diffusion for professional portrait generation from amateur photos faces challenges in maintaining facial resemblance. This paper evaluates the impact of augmentation strategies on two personalization methods: DreamBooth and InstantID. We compare classical augmentations (flipping, cropping, color adjustments) with generative augmentation using InstantID's synthetic images to enrich training data. Using SDXL and a new FaceDistance metric based on FaceNet, we quantitatively assess facial similarity. Results show classical augmentations can cause artifacts harming identity retention, while InstantID improves fidelity when balanced with real images to avoid overfitting. A user study with 97 participants confirms high photorealism and preferences for InstantID's polished look versus DreamBooth's identity accuracy. Our findings inform effective augmentation strategies for personalized text-to-image generation.