QR-LoRA: Efficient and Disentangled Fine-tuning via QR Decomposition for Customized Generation
This addresses the issue of undesired feature entanglement in customized generation for users of text-to-image models, offering a more efficient and disentangled fine-tuning approach.
The paper tackles the problem of feature entanglement in text-to-image models when combining multiple LoRA models for content-style fusion, proposing QR-LoRA, which uses QR decomposition to separate visual attributes and reduces trainable parameters by half compared to conventional LoRA methods.
Existing text-to-image models often rely on parameter fine-tuning techniques such as Low-Rank Adaptation (LoRA) to customize visual attributes. However, when combining multiple LoRA models for content-style fusion tasks, unstructured modifications of weight matrices often lead to undesired feature entanglement between content and style attributes. We propose QR-LoRA, a novel fine-tuning framework leveraging QR decomposition for structured parameter updates that effectively separate visual attributes. Our key insight is that the orthogonal Q matrix naturally minimizes interference between different visual features, while the upper triangular R matrix efficiently encodes attribute-specific transformations. Our approach fixes both Q and R matrices while only training an additional task-specific $ΔR$ matrix. This structured design reduces trainable parameters to half of conventional LoRA methods and supports effective merging of multiple adaptations without cross-contamination due to the strong disentanglement properties between $ΔR$ matrices. Experiments demonstrate that QR-LoRA achieves superior disentanglement in content-style fusion tasks, establishing a new paradigm for parameter-efficient, disentangled fine-tuning in generative models. The project page is available at: https://luna-ai-lab.github.io/QR-LoRA/.