Crowdsourcing Learning as Domain Adaptation: A Case Study on Named Entity Recognition
This work addresses the challenge of leveraging noisy crowdsourced data for NER, offering a novel approach that could benefit NLP practitioners by reducing reliance on expert annotations.
The paper tackled the problem of improving named entity recognition (NER) using crowdsourced annotations by treating them as gold-standard per annotator and applying domain adaptation methods, resulting in state-of-the-art performance on a benchmark dataset and impressive gains with minimal expert annotations.
Crowdsourcing is regarded as one prospective solution for effective supervised learning, aiming to build large-scale annotated training data by crowd workers. Previous studies focus on reducing the influences from the noises of the crowdsourced annotations for supervised models. We take a different point in this work, regarding all crowdsourced annotations as gold-standard with respect to the individual annotators. In this way, we find that crowdsourcing could be highly similar to domain adaptation, and then the recent advances of cross-domain methods can be almost directly applied to crowdsourcing. Here we take named entity recognition (NER) as a study case, suggesting an annotator-aware representation learning model that inspired by the domain adaptation methods which attempt to capture effective domain-aware features. We investigate both unsupervised and supervised crowdsourcing learning, assuming that no or only small-scale expert annotations are available. Experimental results on a benchmark crowdsourced NER dataset show that our method is highly effective, leading to a new state-of-the-art performance. In addition, under the supervised setting, we can achieve impressive performance gains with only a very small scale of expert annotations.