Norm-Bounded Low-Rank Adaptation
This addresses the need for more robust and stable fine-tuning methods in machine learning, particularly for natural language generation and vision applications, though it appears incremental as an enhancement to existing LoRA techniques.
The paper tackles the problem of parameter-efficient fine-tuning by proposing norm-bounded low-rank adaptation (NB-LoRA), which enforces explicit bounds on singular values to satisfy norm constraints, resulting in performance matching or surpassing competing LoRA methods with stronger hyper-parameter robustness and avoidance of catastrophic forgetting in vision tasks.
In this work, we propose norm-bounded low-rank adaptation (NB-LoRA) for parameter-efficient fine tuning. NB-LoRA is a novel parameterization of low-rank weight adaptations that admits explicit bounds on each singular value of the adaptation matrix, which can thereby satisfy any prescribed unitarily invariant norm bound, including the Schatten norms (e.g., nuclear, Frobenius, spectral norm). The proposed parameterization is unconstrained, smooth, and complete, i.e. it covers all matrices satisfying the prescribed rank and singular-value bounds. Natural language generation experiments show that NB-LoRA matches or surpasses performance of competing LoRA methods, while exhibiting stronger hyper-parameter robustness. Vision fine-tuning experiments show that NB-LoRA can avoid model catastrophic forgetting without minor cost on adaptation performance, and compared to existing approaches it is substantially more robust to a hyper-parameters such as including adaptation rank, learning rate and number of training epochs.