Rethinking Efficient Tuning Methods from a Unified Perspective
This work addresses the problem of inefficient adaptation in transfer learning for AI practitioners, offering a more flexible and effective tuning framework, though it is incremental in building upon existing methods.
The paper tackles the limitation of existing parameter-efficient tuning methods that only adjust parts of pre-trained models, proposing a unified framework called U-Tuning that encompasses and extends these methods. It achieves on-par or better performance on CIFAR-100 and FGVC datasets compared to existing approaches.
Parameter-efficient transfer learning (PETL) based on large-scale pre-trained foundation models has achieved great success in various downstream applications. Existing tuning methods, such as prompt, prefix, and adapter, perform task-specific lightweight adjustments to different parts of the original architecture. However, they take effect on only some parts of the pre-trained models, i.e., only the feed-forward layers or the self-attention layers, which leaves the remaining frozen structures unable to adapt to the data distributions of downstream tasks. Further, the existing structures are strongly coupled with the Transformers, hindering parameter-efficient deployment as well as the design flexibility for new approaches. In this paper, we revisit the design paradigm of PETL and derive a unified framework U-Tuning for parameter-efficient transfer learning, which is composed of an operation with frozen parameters and a unified tuner that adapts the operation for downstream applications. The U-Tuning framework can simultaneously encompass existing methods and derive new approaches for parameter-efficient transfer learning, which prove to achieve on-par or better performances on CIFAR-100 and FGVC datasets when compared with existing PETL methods.