End-to-End Learning from Noisy Crowd to Supervised Machine Learning Models
This work provides a method to improve the quality of supervised machine learning models by effectively handling noisy labels from crowd-sourced data, which is a common challenge for data scientists and ML practitioners.
The paper proposes an end-to-end learning framework that combines deep models and human experts to learn from noisy crowd-sourced data. Their on-line label aggregation method, which estimates annotator confusion matrices, reduces label error rates by over 30%, and relabeling just 10% of the data with an expert achieves over 90% classification accuracy with SVM.
Labeling real-world datasets is time consuming but indispensable for supervised machine learning models. A common solution is to distribute the labeling task across a large number of non-expert workers via crowd-sourcing. Due to the varying background and experience of crowd workers, the obtained labels are highly prone to errors and even detrimental to the learning models. In this paper, we advocate using hybrid intelligence, i.e., combining deep models and human experts, to design an end-to-end learning framework from noisy crowd-sourced data, especially in an on-line scenario. We first summarize the state-of-the-art solutions that address the challenges of noisy labels from non-expert crowd and learn from multiple annotators. We show how label aggregation can benefit from estimating the annotators' confusion matrices to improve the learning process. Moreover, with the help of an expert labeler as well as classifiers, we cleanse aggregated labels of highly informative samples to enhance the final classification accuracy. We demonstrate the effectiveness of our strategies on several image datasets, i.e. UCI and CIFAR-10, using SVM and deep neural networks. Our evaluation shows that our on-line label aggregation with confusion matrix estimation reduces the error rate of labels by over 30%. Furthermore, relabeling only 10% of the data using the expert's results in over 90% classification accuracy with SVM.