Unveiling the Multi-Annotation Process: Examining the Influence of Annotation Quantity and Instance Difficulty on Model Performance
This challenges a common belief in NLP about dataset construction, potentially influencing how researchers design and evaluate multi-annotator datasets.
The paper investigates how varying annotation quantities and instance difficulty affect model performance in NLP, finding that multi-annotation datasets do not always outperform single-annotation ones, with performance gains varying based on annotation budget.
The NLP community has long advocated for the construction of multi-annotator datasets to better capture the nuances of language interpretation, subjectivity, and ambiguity. This paper conducts a retrospective study to show how performance scores can vary when a dataset expands from a single annotation per instance to multiple annotations. We propose a novel multi-annotator simulation process to generate datasets with varying annotation budgets. We show that similar datasets with the same annotation budget can lead to varying performance gains. Our findings challenge the popular belief that models trained on multi-annotation examples always lead to better performance than models trained on single or few-annotation examples.