ColA: Collaborative Adaptation with Gradient Learning
This addresses computational overheads in FTaaS for numerous users, offering a more cost-effective solution, though it appears incremental as an enhancement to PEFT methods.
The paper tackles the high computational cost of fine-tuning large models, especially in Fine-Tuning as a Service (FTaaS), by introducing ColA, a parameter-free, model-agnostic approach that decouples gradient computations to offload work to low-cost devices, achieving performance on par or better than existing PEFT methods on benchmarks.
A primary function of back-propagation is to compute both the gradient of hidden representations and parameters for optimization with gradient descent. Training large models requires high computational costs due to their vast parameter sizes. While Parameter-Efficient Fine-Tuning (PEFT) methods aim to train smaller auxiliary models to save computational space, they still present computational overheads, especially in Fine-Tuning as a Service (FTaaS) for numerous users. We introduce Collaborative Adaptation (ColA) with Gradient Learning (GL), a parameter-free, model-agnostic fine-tuning approach that decouples the computation of the gradient of hidden representations and parameters. In comparison to PEFT methods, ColA facilitates more cost-effective FTaaS by offloading the computation of the gradient to low-cost devices. We also provide a theoretical analysis of ColA and experimentally demonstrate that ColA can perform on par or better than existing PEFT methods on various benchmarks.