Efficient Hierarchical Domain Adaptation for Pretrained Language Models
This work addresses the challenge of domain adaptation for large language models, enabling more efficient and effective transfer to diverse domains, which is incremental as it builds on existing adapter approaches.
The paper tackles the problem of adapting pretrained language models to multiple domains efficiently by introducing a hierarchical adapter method that shares parameters among related domains to avoid negative interference, resulting in across-the-board improvements in-domain with GPT-2 on a large subset of C4 websites and further gains in generalization for held-out domains at marginal inference cost.
The remarkable success of large language models has been driven by dense models trained on massive unlabeled, unstructured corpora. These corpora typically contain text from diverse, heterogeneous sources, but information about the source of the text is rarely used during training. Transferring their knowledge to a target domain is typically done by continuing training in-domain. In this paper, we introduce a method to permit domain adaptation to many diverse domains using a computationally efficient adapter approach. Our method is based on the observation that textual domains are partially overlapping, and we represent domains as a hierarchical tree structure where each node in the tree is associated with a set of adapter weights. When combined with a frozen pretrained language model, this approach enables parameter sharing among related domains, while avoiding negative interference between unrelated ones. Experimental results with GPT-2 and a large fraction of the 100 most represented websites in C4 show across-the-board improvements in-domain. We additionally provide an inference time algorithm for a held-out domain and show that averaging over multiple paths through the tree enables further gains in generalization, while adding only a marginal cost to inference.