$α$-LoRA: Effective Fine-Tuning via Base Model Rescaling
This addresses the need for more effective fine-tuning methods in transfer learning, though it appears incremental as it builds on existing reparameterization approaches like LoRA.
The paper tackles the problem of improving generalization in fine-tuned models by introducing a new class of reparameterization methods, with results validated through theoretical analysis and experiments including LLM fine-tuning.
Fine-tuning has proven to be highly effective in adapting pre-trained models to perform better on new desired tasks with minimal data samples. Among the most widely used approaches are reparameterization methods, which update a target module by augmenting its frozen weight matrix with an additional trainable weight matrix. The most prominent example is Low Rank Adaption (LoRA), which gained significant attention in recent years. In this paper, we introduce a new class of reparameterization methods for transfer learning, designed to enhance the generalization ability of fine-tuned models. We establish the effectiveness of our approach in a high-dimensional binary classification setting using tools from Random Matrix Theory, and further validate our theoretical findings through more realistic experiments, such as fine-tuning LLMs.