Pre-train or Annotate? Domain Adaptation with a Constrained Budget
This work addresses a practical problem for NLP practitioners by providing an economical strategy for domain adaptation under budget constraints, though it is incremental as it builds on existing pre-training and annotation methods.
The paper tackles the problem of maximizing performance in domain adaptation for NLP under a fixed budget by comparing the costs and benefits of data annotation versus in-domain pre-training. It finds that for small budgets, spending all on annotation is best, while for larger budgets, a combination of annotation and pre-training yields better performance, with experiments conducted on three procedural text datasets.
Recent work has demonstrated that pre-training in-domain language models can boost performance when adapting to a new domain. However, the costs associated with pre-training raise an important question: given a fixed budget, what steps should an NLP practitioner take to maximize performance? In this paper, we view domain adaptation with a constrained budget as a consumer choice problem, where the goal is to select an optimal combination of data annotation and pre-training. We measure annotation costs of three procedural text datasets, along with the pre-training costs of several in-domain language models. The utility of different combinations of pre-training and data annotation are evaluated under varying budget constraints to assess which combination strategy works best. We find that for small budgets, spending all funds on annotation leads to the best performance; once the budget becomes large enough, however, a combination of data annotation and in-domain pre-training yields better performance. Our experiments suggest task-specific data annotation should be part of an economical strategy when adapting an NLP model to a new domain.