A Comprehensive Analysis of Adapter Efficiency
This work addresses practitioners in NLP by showing that adapters may not provide efficiency advantages, making it an incremental analysis that challenges common assumptions in the field.
The paper tackled the problem of whether parameter-efficient fine-tuning (PEFT) with adapters leads to efficiency gains in training, deployment, and maintainability for Natural Language Understanding tasks, finding that adapters are relatively expensive to train, have slightly higher deployment latency, and offer no clear benefits over full fine-tuning or multi-task training.
Adapters have been positioned as a parameter-efficient fine-tuning (PEFT) approach, whereby a minimal number of parameters are added to the model and fine-tuned. However, adapters have not been sufficiently analyzed to understand if PEFT translates to benefits in training/deployment efficiency and maintainability/extensibility. Through extensive experiments on many adapters, tasks, and languages in supervised and cross-lingual zero-shot settings, we clearly show that for Natural Language Understanding (NLU) tasks, the parameter efficiency in adapters does not translate to efficiency gains compared to full fine-tuning of models. More precisely, adapters are relatively expensive to train and have slightly higher deployment latency. Furthermore, the maintainability/extensibility benefits of adapters can be achieved with simpler approaches like multi-task training via full fine-tuning, which also provide relatively faster training times. We, therefore, recommend that for moderately sized models for NLU tasks, practitioners should rely on full fine-tuning or multi-task training rather than using adapters. Our code is available at https://github.com/AI4Bharat/adapter-efficiency.