Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction
This work addresses the challenge of expensive expert annotations in specialized domains like biomedicine by optimizing annotation strategies, though it is incremental as it builds on existing difficulty modeling and routing approaches.
The paper tackled the problem of high annotation costs in biomedical NLP by modeling instance difficulty to improve model performance and task routing, showing that excluding difficult instances, weighting them during training, and routing them to experts yields consistent gains, with experiments on a biomedical information extraction task demonstrating these improvements.
Modern NLP systems require high-quality annotated data. In specialized domains, expert annotations may be prohibitively expensive. An alternative is to rely on crowdsourcing to reduce costs at the risk of introducing noise. In this paper we demonstrate that directly modeling instance difficulty can be used to improve model performance, and to route instances to appropriate annotators. Our difficulty prediction model combines two learned representations: a `universal' encoder trained on out-of-domain data, and a task-specific encoder. Experiments on a complex biomedical information extraction task using expert and lay annotators show that: (i) simply excluding from the training data instances predicted to be difficult yields a small boost in performance; (ii) using difficulty scores to weight instances during training provides further, consistent gains; (iii) assigning instances predicted to be difficult to domain experts is an effective strategy for task routing. Our experiments confirm the expectation that for specialized tasks expert annotations are higher quality than crowd labels, and hence preferable to obtain if practical. Moreover, augmenting small amounts of expert data with a larger set of lay annotations leads to further improvements in model performance.