Effective Use of Bidirectional Language Modeling for Transfer Learning in Biomedical Named Entity Recognition
This addresses the data scarcity issue for researchers and practitioners in biomedical text mining, though it is incremental as it adapts existing transfer learning methods to this domain.
The paper tackled the problem of limited labeled data in biomedical named entity recognition by using a bidirectional language model pretrained on unlabeled text to initialize an NER model, resulting in substantial F1 score improvements on four benchmark datasets and faster training with fewer examples.
Biomedical named entity recognition (NER) is a fundamental task in text mining of medical documents and has many applications. Deep learning based approaches to this task have been gaining increasing attention in recent years as their parameters can be learned end-to-end without the need for hand-engineered features. However, these approaches rely on high-quality labeled data, which is expensive to obtain. To address this issue, we investigate how to use unlabeled text data to improve the performance of NER models. Specifically, we train a bidirectional language model (BiLM) on unlabeled data and transfer its weights to "pretrain" an NER model with the same architecture as the BiLM, which results in a better parameter initialization of the NER model. We evaluate our approach on four benchmark datasets for biomedical NER and show that it leads to a substantial improvement in the F1 scores compared with the state-of-the-art approaches. We also show that BiLM weight transfer leads to a faster model training and the pretrained model requires fewer training examples to achieve a particular F1 score.