Parameter-efficient Model Adaptation for Vision Transformers
This work addresses the need for efficient model adaptation in computer vision, offering a practical solution for resource-constrained applications, though it is incremental as it builds on existing parameter-efficient methods.
The paper tackles the problem of efficiently adapting large pretrained vision transformers to downstream image classification tasks by proposing a parameter-efficient adaptation framework that selects submodules and projects them into a subspace using Kronecker Adaptation, achieving the best tradeoff between accuracy and parameter efficiency across 27 datasets in few-shot and full-shot settings.
In computer vision, it has achieved great transfer learning performance via adapting large-scale pretrained vision models (e.g., vision transformers) to downstream tasks. Common approaches for model adaptation either update all model parameters or leverage linear probes. In this paper, we aim to study parameter-efficient model adaptation strategies for vision transformers on the image classification task. We formulate efficient model adaptation as a subspace training problem and perform a comprehensive benchmarking over different efficient adaptation methods. We conduct an empirical study on each efficient model adaptation method focusing on its performance alongside parameter cost. Furthermore, we propose a parameter-efficient model adaptation framework, which first selects submodules by measuring local intrinsic dimensions and then projects them into subspace for further decomposition via a novel Kronecker Adaptation (KAdaptation) method. We analyze and compare our method with a diverse set of baseline model adaptation methods (including state-of-the-art methods for pretrained language models). Our method performs the best in terms of the tradeoff between accuracy and parameter efficiency across 20 image classification datasets under the few-shot setting and 7 image classification datasets under the full-shot setting.