Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functions
This work addresses parameter growth issues in fine-tuning large models, offering a more efficient adaptation method for machine learning practitioners.
The paper tackles the parameter inefficiency of standard Low-Rank Adaptation (LoRA) by proposing GenLoRA, which replaces explicit basis vectors with lightweight nonlinear radial basis functions to generate them, achieving higher effective ranks and superior fine-tuning performance across multiple datasets and architectures.
Low-rank adaptation (LoRA) approximates the update of a pretrained weight matrix using the product of two low-rank matrices. However, standard LoRA follows an explicit-rank paradigm, where increasing model capacity requires adding more rows or columns (i.e., basis vectors) to the low-rank matrices, leading to substantial parameter growth. In this paper, we find that these basis vectors exhibit significant parameter redundancy and can be compactly represented by lightweight nonlinear functions. Therefore, we propose Generative Low-Rank Adapter (GenLoRA), which replaces explicit basis vector storage with nonlinear basis vector generation. Specifically, GenLoRA maintains a latent vector for each low-rank matrix and employs a set of lightweight radial basis functions (RBFs) to synthesize the basis vectors. Each RBF requires far fewer parameters than an explicit basis vector, enabling higher parameter efficiency in GenLoRA. Extensive experiments across multiple datasets and architectures show that GenLoRA attains higher effective LoRA ranks under smaller parameter budgets, resulting in superior fine-tuning performance. The code is available at https://anonymous.4open.science/r/GenLoRA-1519.