Robust Deep Learning from Crowds with Belief Propagation
This work addresses the challenge of improving deep learning accuracy from noisy crowdsourced data, which is incremental as it builds upon existing Bayesian methods with a more sophisticated approximation.
The paper tackles the problem of noisy labels in crowdsourced datasets by developing a neural-powered Bayesian framework, resulting in deepBP, which shows improved robustness against wrong priors, feature overfitting, and extreme workers compared to existing methods.
Crowdsourcing systems enable us to collect large-scale dataset, but inherently suffer from noisy labels of low-paid workers. We address the inference and learning problems using such a crowdsourced dataset with noise. Due to the nature of sparsity in crowdsourcing, it is critical to exploit both probabilistic model to capture worker prior and neural network to extract task feature despite risks from wrong prior and overfitted feature in practice. We hence establish a neural-powered Bayesian framework, from which we devise deepMF and deepBP with different choice of variational approximation methods, mean field (MF) and belief propagation (BP), respectively. This provides a unified view of existing methods, which are special cases of deepMF with different priors. In addition, our empirical study suggests that deepBP is a new approach, which is more robust against wrong prior, feature overfitting and extreme workers thanks to the more sophisticated BP than MF.