LERT: A Linguistically-motivated Pre-trained Language Model
This work addresses the need for linguistically-informed pre-training in natural language processing, offering a domain-specific enhancement for Chinese language tasks.
The authors tackled the problem of enriching pre-trained language models with linguistic features by proposing LERT, which integrates three types of linguistic features with masked language model pre-training, and it achieved significant improvements over baselines on ten Chinese NLU tasks.
Pre-trained Language Model (PLM) has become a representative foundation model in the natural language processing field. Most PLMs are trained with linguistic-agnostic pre-training tasks on the surface form of the text, such as the masked language model (MLM). To further empower the PLMs with richer linguistic features, in this paper, we aim to propose a simple but effective way to learn linguistic features for pre-trained language models. We propose LERT, a pre-trained language model that is trained on three types of linguistic features along with the original MLM pre-training task, using a linguistically-informed pre-training (LIP) strategy. We carried out extensive experiments on ten Chinese NLU tasks, and the experimental results show that LERT could bring significant improvements over various comparable baselines. Furthermore, we also conduct analytical experiments in various linguistic aspects, and the results prove that the design of LERT is valid and effective. Resources are available at https://github.com/ymcui/LERT