VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding
This addresses the issue of pre-trained language models underperforming in specific domains due to limited corpus, offering a solution for domain adaptation with incremental improvements.
The paper tackles the problem of domain-adaptive language understanding by proposing VarMAE, a Transformer-based model with a context uncertainty learning module, which achieves efficient adaptation to new domains with limited resources as demonstrated on science- and finance-domain NLU tasks.
Pre-trained language models have achieved promising performance on general benchmarks, but underperform when migrated to a specific domain. Recent works perform pre-training from scratch or continual pre-training on domain corpora. However, in many specific domains, the limited corpus can hardly support obtaining precise representations. To address this issue, we propose a novel Transformer-based language model named VarMAE for domain-adaptive language understanding. Under the masked autoencoding objective, we design a context uncertainty learning module to encode the token's context into a smooth latent distribution. The module can produce diverse and well-formed contextual representations. Experiments on science- and finance-domain NLU tasks demonstrate that VarMAE can be efficiently adapted to new domains with limited resources.