Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristics
This work addresses domain adaptation for NLP practitioners, but it is incremental as it builds on existing masking techniques.
The paper tackled the problem of adapting language models to specialized domains by introducing a selective masking method based on genre and topical characteristics, achieving improved performance on the LegalGLUE benchmark in the legal domain.
Recent advances in pre-trained language modeling have facilitated significant progress across various natural language processing (NLP) tasks. Word masking during model training constitutes a pivotal component of language modeling in architectures like BERT. However, the prevalent method of word masking relies on random selection, potentially disregarding domain-specific linguistic attributes. In this article, we introduce an innovative masking approach leveraging genre and topicality information to tailor language models to specialized domains. Our method incorporates a ranking process that prioritizes words based on their significance, subsequently guiding the masking procedure. Experiments conducted using continual pre-training within the legal domain have underscored the efficacy of our approach on the LegalGLUE benchmark in the English language. Pre-trained language models and code are freely available for use.