Generalized Tensor-based Parameter-Efficient Fine-Tuning via Lie Group Transformations
This work addresses the problem of adapting pre-trained models efficiently for diverse tasks, offering a novel generalization that is incremental but extends applicability to higher-dimensional parameter spaces.
The paper tackles the limitation of parameter-efficient fine-tuning (PEFT) methods, which are designed for linear layers and ignore higher-dimensional parameter spaces like convolutional kernels, by proposing a generalization that extends matrix-based PEFT methods to higher-dimensional spaces using Lie group transformations, resulting in clear improvements over existing methods in computer vision and natural language processing experiments.
Adapting pre-trained foundation models for diverse downstream tasks is a core practice in artificial intelligence. However, the wide range of tasks and high computational costs make full fine-tuning impractical. To overcome this, parameter-efficient fine-tuning (PEFT) methods like LoRA have emerged and are becoming a growing research focus. Despite the success of these methods, they are primarily designed for linear layers, focusing on two-dimensional matrices while largely ignoring higher-dimensional parameter spaces like convolutional kernels. Moreover, directly applying these methods to higher-dimensional parameter spaces often disrupts their structural relationships. Given the rapid advancements in matrix-based PEFT methods, rather than designing a specialized strategy, we propose a generalization that extends matrix-based PEFT methods to higher-dimensional parameter spaces without compromising their structural properties. Specifically, we treat parameters as elements of a Lie group, with updates modeled as perturbations in the corresponding Lie algebra. These perturbations are mapped back to the Lie group through the exponential map, ensuring smooth, consistent updates that preserve the inherent structure of the parameter space. Extensive experiments on computer vision and natural language processing validate the effectiveness and versatility of our approach, demonstrating clear improvements over existing methods.