Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
This addresses the challenge of deploying large models in specific domains with latency and capacity constraints, offering an incremental improvement through adaptation and distillation techniques.
The paper tackles the problem of domain shift and inefficiency when applying large pre-trained language models to specific domains by proposing an approach to develop smaller, faster models, achieving better performance than BERT BASE while being 3.3x smaller and 5.1x faster in biomedical and computer science tasks.
Large pre-trained models have achieved great success in many natural language processing tasks. However, when they are applied in specific domains, these models suffer from domain shift and bring challenges in fine-tuning and online serving for latency and capacity constraints. In this paper, we present a general approach to developing small, fast and effective pre-trained models for specific domains. This is achieved by adapting the off-the-shelf general pre-trained models and performing task-agnostic knowledge distillation in target domains. Specifically, we propose domain-specific vocabulary expansion in the adaptation stage and employ corpus level occurrence probability to choose the size of incremental vocabulary automatically. Then we systematically explore different strategies to compress the large pre-trained models for specific domains. We conduct our experiments in the biomedical and computer science domain. The experimental results demonstrate that our approach achieves better performance over the BERT BASE model in domain-specific tasks while 3.3x smaller and 5.1x faster than BERT BASE. The code and pre-trained models are available at https://aka.ms/adalm.