Towards Symmetric Low-Rank Adapters
This is an incremental improvement for efficient fine-tuning in machine learning, benefiting practitioners by reducing computational and storage costs.
The paper tackles the problem of reducing the number of weights in low-rank adapters for fine-tuning pre-trained models by introducing SymLoRA, which uses symmetric low-rank weight matrices to cut finetuning weights by approximately half with negligible losses in downstream task performance.
In this paper, we introduce Symmetric Low-Rank Adapters, an optimized variant of LoRA with even fewer weights. This method utilizes Low-Rank Symmetric Weight Matrices to learn downstream tasks more efficiently. Traditional LoRA accumulates fine-tuning weights with the original pre-trained weights via a Singular Value Decomposition (SVD) like approach, i.e., model weights are fine-tuned via updates of the form $BA$ (where $B \in \mathbb{R}^{n\times r}$, $A \in \mathbb{R}^{r\times n}$, and $r$ is the rank of the merged weight matrix). In contrast, our approach, named SymLoRA, represents fine-tuning weights as a Spectral Decomposition, i.e., $Q \, diag(Λ)\, Q^T$, where $Q \in \mathbb{R}^{n\times r}$ and $Λ\in \mathbb{R}^r$. SymLoRA requires approximately half of the finetuning weights. Here, we show that this approach has negligible losses in downstream efficacy.