Zero-Shot Adaptation of Parameter-Efficient Fine-Tuning in Diffusion Models
This addresses the challenge of data constraints in fine-tuning for text-to-image models, though it appears incremental as it builds on existing parameter-efficient methods like LoRA.
The paper tackles the problem of adapting parameter-efficient fine-tuning in diffusion models without retraining by introducing ProLoRA, which transfers pre-trained low-rank adjustments from a source to a target model via projection, achieving successful knowledge transfer and comparable performance.
We introduce ProLoRA, enabling zero-shot adaptation of parameter-efficient fine-tuning in text-to-image diffusion models. ProLoRA transfers pre-trained low-rank adjustments (e.g., LoRA) from a source to a target model without additional training data. This overcomes the limitations of traditional methods that require retraining when switching base models, often challenging due to data constraints. ProLoRA achieves this via projection of source adjustments into the target model's weight space, leveraging subspace and null space similarities and selectively targeting aligned layers. Evaluations on established text-to-image models demonstrate successful knowledge transfer and comparable performance without retraining.