PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models
This addresses domain adaptation for NLP practitioners by enhancing model generalization across domains, though it is incremental as it builds on existing pivot-based methods.
The paper tackles the problem of domain adaptation for NLP by proposing PERL, a model that extends contextualized embeddings like BERT with pivot-based fine-tuning, incorporating massive unlabeled corpora beyond source and target domains. It outperforms baselines across 22 sentiment classification setups, improving in-domain performance, model stability, and enabling effective reduced-size models.
Pivot-based neural representation models have lead to significant progress in domain adaptation for NLP. However, previous works that follow this approach utilize only labeled data from the source domain and unlabeled data from the source and target domains, but neglect to incorporate massive unlabeled corpora that are not necessarily drawn from these domains. To alleviate this, we propose PERL: A representation learning model that extends contextualized word embedding models such as BERT with pivot-based fine-tuning. PERL outperforms strong baselines across 22 sentiment classification domain adaptation setups, improves in-domain model performance, yields effective reduced-size models and increases model stability.