VectorFit : Adaptive Singular & Bias Vector Fine-Tuning of Pre-trained Foundation Models
This addresses the problem of efficient fine-tuning for large foundation models, offering a more parameter-efficient method that reduces the gap to full fine-tuning, though it is incremental in improving existing PEFT approaches.
The paper tackles the performance gap between parameter-efficient fine-tuning (PEFT) methods and full fine-tuning by introducing VectorFit, which adaptively trains singular vectors and biases of pre-trained weights to achieve high-rank incremental matrices. It shows superior results with 9× fewer trainable parameters than leading PEFT methods across 19 language and vision datasets.
Popular PEFT methods reduce trainable parameter count for fine-tuning by parameterizing new low-rank or sparse trainable weights in parallel to the frozen pre-trained weights $W$. However, these weights are trained from scratch, and there exists a performance gap between these methods and full fine-tuning, especially in low-budget settings. We introduce VectorFit, a new way of parameterization that efficiently utilizes the existing knowledge embedded in $W$ by adaptively training their singular vectors and biases. We show that utilizing the structural and transformational properties of $W$ in this way can lead to high-rank incremental weight matrices $ΔW$, comparable to that of full fine-tuning. VectorFit delivers superior results with 9$\boldsymbol\times$ fewer trainable parameters than the leading PEFT methods. Through comprehensive experiments across 19 datasets covering a wide range of language and vision tasks such as natural language understanding and generation, question answering, image classification, and image generation, we demonstrate that VectorFit surpasses baselines in terms of performance as a function of parameter-efficiency.