Focusing on Potential Named Entities During Active Label Acquisition
This work addresses the high annotation cost problem for domain-specific NER applications, though it is incremental as it builds on existing active learning frameworks.
The authors tackled the challenge of imbalanced class distribution in active learning for named entity recognition by proposing new query evaluation functions that focus on potential positive tokens and a data-driven normalization approach. Their method reduced the number of annotated tokens by up to 30% while achieving comparable or better performance on three domain-specific datasets.
Named entity recognition (NER) aims to identify mentions of named entities in an unstructured text and classify them into predefined named entity classes. While deep learning-based pre-trained language models help to achieve good predictive performances in NER, many domain-specific NER applications still call for a substantial amount of labeled data. Active learning (AL), a general framework for the label acquisition problem, has been used for NER tasks to minimize the annotation cost without sacrificing model performance. However, the heavily imbalanced class distribution of tokens introduces challenges in designing effective AL querying methods for NER. We propose several AL sentence query evaluation functions that pay more attention to potential positive tokens, and evaluate these proposed functions with both sentence-based and token-based cost evaluation strategies. We also propose a better data-driven normalization approach to penalize sentences that are too long or too short. Our experiments on three datasets from different domains reveal that the proposed approach reduces the number of annotated tokens while achieving better or comparable prediction performance with conventional methods.