Block-wise LoRA: Revisiting Fine-grained LoRA for Effective Personalization and Stylization in Text-to-Image Generation
This addresses the challenge for users and developers in creating customized and stylized images using diffusion models, though it appears incremental as it builds on existing LoRA methods.
The paper tackles the problem of achieving effective personalization and stylization in text-to-image generation by proposing block-wise LoRA, a fine-grained fine-tuning method that improves image faithfulness to prompts, target identity, and desired style.
The objective of personalization and stylization in text-to-image is to instruct a pre-trained diffusion model to analyze new concepts introduced by users and incorporate them into expected styles. Recently, parameter-efficient fine-tuning (PEFT) approaches have been widely adopted to address this task and have greatly propelled the development of this field. Despite their popularity, existing efficient fine-tuning methods still struggle to achieve effective personalization and stylization in T2I generation. To address this issue, we propose block-wise Low-Rank Adaptation (LoRA) to perform fine-grained fine-tuning for different blocks of SD, which can generate images faithful to input prompts and target identity and also with desired style. Extensive experiments demonstrate the effectiveness of the proposed method.