AC-LoRA: Auto Component LoRA for Personalized Artistic Style Image Generation
This work addresses a specific bottleneck in fine-tuning large text-to-image models for personalization, offering an incremental improvement over existing LoRA methods.
The paper tackles the challenge of adjusting the rank parameter in LoRA-based methods for personalized artistic style image generation by proposing AC-LoRA, which automatically separates signal and noise components using SVD and dynamic heuristics, resulting in a 9% average improvement in metrics like FID and CLIP.
Personalized image generation allows users to preserve styles or subjects of a provided small set of images for further image generation. With the advancement in large text-to-image models, many techniques have been developed to efficiently fine-tune those models for personalization, such as Low Rank Adaptation (LoRA). However, LoRA-based methods often face the challenge of adjusting the rank parameter to achieve satisfactory results. To address this challenge, AutoComponent-LoRA (AC-LoRA) is proposed, which is able to automatically separate the signal component and noise component of the LoRA matrices for fast and efficient personalized artistic style image generation. This method is based on Singular Value Decomposition (SVD) and dynamic heuristics to update the hyperparameters during training. Superior performance over existing methods in overcoming model underfitting or overfitting problems is demonstrated. The results were validated using FID, CLIP, DINO, and ImageReward, achieving an average of 9% improvement.