Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation
This work addresses parameter efficiency in fine-tuning large pretrained models for downstream tasks, representing an incremental improvement over existing OFT methods.
The paper tackles the low parameter efficiency and limited adaptation capability of Orthogonal Fine-Tuning (OFT) by proposing quasi-Givens Orthogonal Fine-Tuning (qGOFT), which reduces parameter complexity from O(d^2) to O(d) using Givens rotations and enhances adaptation with norm and angular adjustments.
With the increasingly powerful performances and enormous scales of pretrained models, promoting parameter efficiency in fine-tuning has become a crucial need for effective and efficient adaptation to various downstream tasks. One representative line of fine-tuning methods is Orthogonal Fine-tuning (OFT), which rigorously preserves the angular distances within the parameter space to preserve the pretrained knowledge. Despite the empirical effectiveness, OFT still suffers low parameter efficiency at $\mathcal{O}(d^2)$ and limited capability of downstream adaptation. Inspired by Givens rotation, in this paper, we proposed quasi-Givens Orthogonal Fine-Tuning (qGOFT) to address the problems. We first use $\mathcal{O}(d)$ Givens rotations to accomplish arbitrary orthogonal transformation in $SO(d)$ with provable equivalence, reducing parameter complexity from $\mathcal{O}(d^2)$ to $\mathcal{O}(d)$. Then we introduce flexible norm and relative angular adjustments under soft orthogonality regularization to enhance the adaptation capability of downstream semantic deviations. Extensive experiments on various tasks and pretrained models validate the effectiveness of our methods.