BiLoRA: A Bi-level Optimization Framework for Overfitting-Resilient Low-Rank Adaptation of Large Pre-trained Models
This addresses overfitting issues in fine-tuning for downstream tasks, offering a more resilient approach for users of large models, though it is incremental as it builds on existing LoRA techniques.
The paper tackles overfitting in low-rank adaptation (LoRA) methods for fine-tuning large pre-trained models by introducing BiLoRA, a bi-level optimization framework that splits training across data subsets, resulting in significant performance improvements on ten datasets for natural language tasks compared to LoRA and other methods with similar parameter counts.
Low-rank adaptation (LoRA) is a popular method for fine-tuning large-scale pre-trained models in downstream tasks by learning low-rank incremental matrices. Though LoRA and its variants effectively reduce the number of trainable parameters compared to full fine-tuning methods, they often overfit training data, resulting in sub-optimal generalization on test data. To address this problem, we introduce BiLoRA, an overfitting-alleviating fine-tuning approach based on bi-level optimization (BLO). BiLoRA employs pseudo singular value decomposition to parameterize low-rank incremental matrices and splits the training of pseudo singular vectors and values across two different subsets of training data. This division, embedded within separate levels of the BLO framework, mitigates the risk of overfitting to a single dataset. Tested on ten datasets covering natural language understanding and generation tasks and applied to various well-known large pre-trained models, BiLoRA significantly outperforms LoRA methods and other fine-tuning approaches, with similar amounts of trainable parameters.