IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
This work addresses inference efficiency for deploying fine-tuned diffusion models, particularly at low bit-widths, but is incremental as it builds on existing LoRA methods.
The paper tackles the problem of fine-tuning quantized diffusion models efficiently by introducing IntLoRA, which adapts models with integer-type low-rank parameters to avoid post-training quantization and performance drops, achieving significant speedup in training and inference without losing performance.
Fine-tuning pre-trained diffusion models under limited budgets has gained great success. In particular, the recent advances that directly fine-tune the quantized weights using Low-rank Adaptation (LoRA) further reduces training costs. Despite these progress, we point out that existing adaptation recipes are not inference-efficient. Specifically, additional post-training quantization (PTQ) on tuned weights is needed during deployment, which results in noticeable performance drop when the bit-width is low. Based on this observation, we introduce IntLoRA, which adapts quantized diffusion models with integer-type low-rank parameters, to include inference efficiency during tuning. Specifically, IntLoRA enables pre-trained weights to remain quantized during training, facilitating fine-tuning on consumer-level GPUs. During inference, IntLoRA weights can be seamlessly merged into pre-trained weights to directly obtain quantized downstream weights without PTQ. Extensive experiments show our IntLoRA achieves significant speedup on both training and inference without losing performance.