NER-BERT: A Pre-trained Model for Low-Resource Entity Tagging
This addresses the data scarcity issue in NER for low-resource domains, representing an incremental improvement through task-specific pre-training.
The paper tackles the problem of poor performance in named entity recognition (NER) models under low-resource conditions by pre-training a NER-BERT model on a newly constructed high-quality NER corpus, resulting in significant outperformance over BERT and other baselines across nine diverse domains.
Named entity recognition (NER) models generally perform poorly when large training datasets are unavailable for low-resource domains. Recently, pre-training a large-scale language model has become a promising direction for coping with the data scarcity issue. However, the underlying discrepancies between the language modeling and NER task could limit the models' performance, and pre-training for the NER task has rarely been studied since the collected NER datasets are generally small or large but with low quality. In this paper, we construct a massive NER corpus with a relatively high quality, and we pre-train a NER-BERT model based on the created dataset. Experimental results show that our pre-trained model can significantly outperform BERT as well as other strong baselines in low-resource scenarios across nine diverse domains. Moreover, a visualization of entity representations further indicates the effectiveness of NER-BERT for categorizing a variety of entities.