Pretrained language model transfer on neural named entity recognition in Indonesian conversational texts
This work addresses the problem of low-resource NER for Indonesian conversational texts, which is incremental as it applies existing transfer learning methods to a new domain and language.
The paper tackled named entity recognition in Indonesian conversational texts with limited labeled data by using pretrained language model transfer, achieving a 32-point absolute improvement in F1 score over baselines when trained on only 100 sentences.
Named entity recognition (NER) is an important task in NLP, which is all the more challenging in conversational domain with their noisy facets. Moreover, conversational texts are often available in limited amount, making supervised tasks infeasible. To learn from small data, strong inductive biases are required. Previous work relied on hand-crafted features to encode these biases until transfer learning emerges. Here, we explore a transfer learning method, namely language model pretraining, on NER task in Indonesian conversational texts. We utilize large unlabeled data (generic domain) to be transferred to conversational texts, enabling supervised training on limited in-domain data. We report two transfer learning variants, namely supervised model fine-tuning and unsupervised pretrained LM fine-tuning. Our experiments show that both variants outperform baseline neural models when trained on small data (100 sentences), yielding an absolute improvement of 32 points of test F1 score. Furthermore, we find that the pretrained LM encodes part-of-speech information which is a strong predictor for NER.