Neighborhood-based Pooling for Population-level Label Distribution Learning
This addresses the challenge of handling annotator disagreement as proper variation rather than noise in supervised learning, though it appears incremental as it builds on existing label-sharing methods.
The paper tackles the problem of population-level label distribution learning (PLDL) where small annotation samples per data item are insufficient to represent population beliefs, proposing an algorithmic framework with statistical tests for sampling size and new neighborhood-based pooling methods for label sharing.
Supervised machine learning often requires human-annotated data. While annotator disagreement is typically interpreted as evidence of noise, population-level label distribution learning (PLDL) treats the collection of annotations for each data item as a sample of the opinions of a population of human annotators, among whom disagreement may be proper and expected, even with no noise present. From this perspective, a typical training set may contain a large number of very small-sized samples, one for each data item, none of which, by itself, is large enough to be considered representative of the underlying population's beliefs about that item. We propose an algorithmic framework and new statistical tests for PLDL that account for sampling size. We apply them to previously proposed methods for sharing labels across similar data items. We also propose new approaches for label sharing, which we call neighborhood-based pooling.