Revisit Parameter-Efficient Transfer Learning: A Two-Stage Paradigm
This work addresses the issue of task distribution shift for practitioners using large pre-trained models, offering an incremental improvement in parameter-efficient transfer learning.
The paper tackles the problem of task distribution shift in parameter-efficient transfer learning by introducing a two-stage paradigm that first aligns the pre-trained model to the target distribution and then adapts task-relevant channels, achieving state-of-the-art performance with an average accuracy on 19 downstream tasks.
Parameter-Efficient Transfer Learning (PETL) aims at efficiently adapting large models pre-trained on massive data to downstream tasks with limited task-specific data. In view of the practicality of PETL, previous works focus on tuning a small set of parameters for each downstream task in an end-to-end manner while rarely considering the task distribution shift issue between the pre-training task and the downstream task. This paper proposes a novel two-stage paradigm, where the pre-trained model is first aligned to the target distribution. Then the task-relevant information is leveraged for effective adaptation. Specifically, the first stage narrows the task distribution shift by tuning the scale and shift in the LayerNorm layers. In the second stage, to efficiently learn the task-relevant information, we propose a Taylor expansion-based importance score to identify task-relevant channels for the downstream task and then only tune such a small portion of channels, making the adaptation to be parameter-efficient. Overall, we present a promising new direction for PETL, and the proposed paradigm achieves state-of-the-art performance on the average accuracy of 19 downstream tasks.