AdapterFusion: Non-Destructive Task Composition for Transfer Learning
This addresses the challenge of effectively combining knowledge from multiple tasks in transfer learning for natural language understanding, with incremental improvements over existing methods.
The paper tackled the problems of catastrophic forgetting and dataset balancing in sequential fine-tuning and multi-task learning by proposing AdapterFusion, a two-stage algorithm that learns task-specific adapters and combines them non-destructively, resulting in outperformance over traditional strategies on 16 NLU tasks.
Sequential fine-tuning and multi-task learning are methods aiming to incorporate knowledge from multiple tasks; however, they suffer from catastrophic forgetting and difficulties in dataset balancing. To address these shortcomings, we propose AdapterFusion, a new two stage learning algorithm that leverages knowledge from multiple tasks. First, in the knowledge extraction stage we learn task specific parameters called adapters, that encapsulate the task-specific information. We then combine the adapters in a separate knowledge composition step. We show that by separating the two stages, i.e., knowledge extraction and knowledge composition, the classifier can effectively exploit the representations learned from multiple tasks in a non-destructive manner. We empirically evaluate AdapterFusion on 16 diverse NLU tasks, and find that it effectively combines various types of knowledge at different layers of the model. We show that our approach outperforms traditional strategies such as full fine-tuning as well as multi-task learning. Our code and adapters are available at AdapterHub.ml.