DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusion
This addresses the need for efficient and high-fidelity personalization in text-to-image generation for users, though it is incremental as it builds on existing LoRA and diffusion model techniques.
The paper tackles the problem of personalized text-to-image generation by proposing DiffLoRA, a method that uses a diffusion model as a hypernetwork to predict personalized LoRA weights from reference images, enabling zero-shot personalization without test-time fine-tuning. It outperforms existing approaches in benchmarks, achieving time efficiency and high identity fidelity.
Personalized text-to-image generation has gained significant attention for its capability to generate high-fidelity portraits of specific identities conditioned on user-defined prompts. Existing methods typically involve test-time fine-tuning or incorporating an additional pre-trained branch. However, these approaches struggle to simultaneously address efficiency, identity fidelity, and the preservation of the model's original generative capabilities. In this paper, we propose DiffLoRA, an efficient method that leverages the diffusion model as a hypernetwork to predict personalized Low-Rank Adaptation (LoRA) weights based on the reference images. By incorporating these LoRA weights into the off-the-shelf text-to-image model, DiffLoRA enables zero-shot personalization during inference, eliminating the need for post-processing optimization. Moreover, we introduce a novel identity-oriented LoRA weights construction pipeline to facilitate the training process of DiffLoRA. The dataset generated through this pipeline enables DiffLoRA to produce consistently high-quality LoRA weights. Notably, the distinctive properties of the diffusion model enhance the generation of superior weights by employing probabilistic modeling to capture intricate structural patterns and thoroughly explore the weight space. Comprehensive experimental results demonstrate that DiffLoRA outperforms existing personalization approaches across multiple benchmarks, achieving both time efficiency and maintaining identity fidelity throughout the personalization process.