UDApter -- Efficient Domain Adaptation Using Adapters
This work addresses the challenge of making domain adaptation more efficient for NLP practitioners, though it is incremental as it builds on existing adapter and UDA techniques.
The paper tackles the problem of parameter inefficiency in unsupervised domain adaptation (UDA) by proposing two adapter-based methods, which achieve competitive performance in sentiment classification and natural language inference tasks while fine-tuning only a fraction of model parameters, such as outperforming DANN and DSN in sentiment classification and being within 0.85% F1 for NLI.
We propose two methods to make unsupervised domain adaptation (UDA) more parameter efficient using adapters, small bottleneck layers interspersed with every layer of the large-scale pre-trained language model (PLM). The first method deconstructs UDA into a two-step process: first by adding a domain adapter to learn domain-invariant information and then by adding a task adapter that uses domain-invariant information to learn task representations in the source domain. The second method jointly learns a supervised classifier while reducing the divergence measure. Compared to strong baselines, our simple methods perform well in natural language inference (MNLI) and the cross-domain sentiment classification task. We even outperform unsupervised domain adaptation methods such as DANN and DSN in sentiment classification, and we are within 0.85% F1 for natural language inference task, by fine-tuning only a fraction of the full model parameters. We release our code at https://github.com/declare-lab/domadapter