Unsupervised Domain Clusters in Pretrained Language Models
This work addresses the challenge of building domain-specific NLP systems when domain labels are unavailable, offering a data-driven solution that is incremental but practical for applications like machine translation.
The paper tackles the problem of defining and selecting domain-specific data in NLP without domain labels, showing that pretrained language models learn sentence representations that cluster by domains, and proposes data selection methods that outperform an established approach in neural machine translation across five domains, achieving improvements in BLEU scores and selection precision/recall.
The notion of "in-domain data" in NLP is often over-simplistic and vague, as textual data varies in many nuanced linguistic aspects such as topic, style or level of formality. In addition, domain labels are many times unavailable, making it challenging to build domain-specific systems. We show that massive pre-trained language models implicitly learn sentence representations that cluster by domains without supervision -- suggesting a simple data-driven definition of domains in textual data. We harness this property and propose domain data selection methods based on such models, which require only a small set of in-domain monolingual data. We evaluate our data selection methods for neural machine translation across five diverse domains, where they outperform an established approach as measured by both BLEU and by precision and recall of sentence selection with respect to an oracle.