Enhanced Nearest Neighbor Classification for Crowdsourcing
This work addresses label noise in crowdsourcing for machine learning, offering incremental improvements in classification accuracy for applications relying on crowdsourced data.
The paper tackles the problem of noisy labels in crowdsourced data classification by proposing an enhanced nearest neighbor classifier (ENN) with algorithms to estimate worker quality, achieving the same regret as an oracle version with expert data and providing a lower bound on sample size for optimal convergence.
In machine learning, crowdsourcing is an economical way to label a large amount of data. However, the noise in the produced labels may deteriorate the accuracy of any classification method applied to the labelled data. We propose an enhanced nearest neighbor classifier (ENN) to overcome this issue. Two algorithms are developed to estimate the worker quality (which is often unknown in practice): one is to construct the estimate based on the denoised worker labels by applying the $k$NN classifier to the expert data; the other is an iterative algorithm that works even without access to the expert data. Other than strong numerical evidence, our proposed methods are proven to achieve the same regret as its oracle version based on high-quality expert data. As a technical by-product, a lower bound on the sample size assigned to each worker to reach the optimal convergence rate of regret is derived.