CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators
This addresses the challenge of noisy labels in crowdsourced data for machine learning practitioners, though it is incremental as it builds on existing methods with a novel approach.
The paper tackled the problem of inferring consensus labels and quality scores from data labeled by multiple annotators, introducing CROWDLAB, which outperformed existing algorithms like Dawid-Skene/GLAD on real-world multi-annotator image data.
Real-world data for classification is often labeled by multiple annotators. For analyzing such data, we introduce CROWDLAB, a straightforward approach to utilize any trained classifier to estimate: (1) A consensus label for each example that aggregates the available annotations; (2) A confidence score for how likely each consensus label is correct; (3) A rating for each annotator quantifying the overall correctness of their labels. Existing algorithms to estimate related quantities in crowdsourcing often rely on sophisticated generative models with iterative inference. CROWDLAB instead uses a straightforward weighted ensemble. Existing algorithms often rely solely on annotator statistics, ignoring the features of the examples from which the annotations derive. CROWDLAB utilizes any classifier model trained on these features, and can thus better generalize between examples with similar features. On real-world multi-annotator image data, our proposed method provides superior estimates for (1)-(3) than existing algorithms like Dawid-Skene/GLAD.