Unsupervised Domain Adaptation with Adapter
This work addresses the challenge of adapting pre-trained models to new domains without distorting learned knowledge, which is incremental as it builds on existing adapter methods for domain adaptation.
The paper tackles the problem of unsupervised domain adaptation with pre-trained language models by proposing an adapter-based fine-tuning approach that preserves generic knowledge and reduces deployment costs, achieving effective results across different tasks, dataset sizes, and domain similarities in experiments on two benchmark datasets.
Unsupervised domain adaptation (UDA) with pre-trained language models (PrLM) has achieved promising results since these pre-trained models embed generic knowledge learned from various domains. However, fine-tuning all the parameters of the PrLM on a small domain-specific corpus distort the learned generic knowledge, and it is also expensive to deployment a whole fine-tuned PrLM for each domain. This paper explores an adapter-based fine-tuning approach for unsupervised domain adaptation. Specifically, several trainable adapter modules are inserted in a PrLM, and the embedded generic knowledge is preserved by fixing the parameters of the original PrLM at fine-tuning. A domain-fusion scheme is introduced to train these adapters using a mix-domain corpus to better capture transferable features. Elaborated experiments on two benchmark datasets are carried out, and the results demonstrate that our approach is effective with different tasks, dataset sizes, and domain similarities.