Regularized Minimax Conditional Entropy for Crowdsourcing
This addresses the challenge of low-quality labels in crowdsourcing, which is critical for applications relying on accurate data labeling, though it appears incremental as it builds on existing entropy-based methods.
The paper tackles the problem of inferring ground truth from noisy crowdsourced labels by proposing a minimax conditional entropy principle, resulting in a unique probabilistic labeling model that jointly parameterizes worker ability and item difficulty, validated on real datasets with binary, multiclass, or ordinal labels.
There is a rapidly increasing interest in crowdsourcing for data labeling. By crowdsourcing, a large number of labels can be often quickly gathered at low cost. However, the labels provided by the crowdsourcing workers are usually not of high quality. In this paper, we propose a minimax conditional entropy principle to infer ground truth from noisy crowdsourced labels. Under this principle, we derive a unique probabilistic labeling model jointly parameterized by worker ability and item difficulty. We also propose an objective measurement principle, and show that our method is the only method which satisfies this objective measurement principle. We validate our method through a variety of real crowdsourcing datasets with binary, multiclass or ordinal labels.