Cost-Accuracy Aware Adaptive Labeling for Active Learning
This addresses the challenge of cost-effective labeling in real-world settings like crowdsourcing, though it is incremental as it builds on existing active learning frameworks.
The paper tackles the problem of active learning with diverse labelers having varying costs and accuracies, proposing an algorithm that selects instances and labelers to achieve higher generalization accuracy at lower cost, demonstrating state-of-the-art performance on five UCI and a crowdsourcing dataset.
Conventional active learning algorithms assume a single labeler that produces noiseless label at a given, fixed cost, and aim to achieve the best generalization performance for given classifier under a budget constraint. However, in many real settings, different labelers have different labeling costs and can yield different labeling accuracies. Moreover, a given labeler may exhibit different labeling accuracies for different instances. This setting can be referred to as active learning with diverse labelers with varying costs and accuracies, and it arises in many important real settings. It is therefore beneficial to understand how to effectively trade-off between labeling accuracy for different instances, labeling costs, as well as the informativeness of training instances, so as to achieve the best generalization performance at the lowest labeling cost. In this paper, we propose a new algorithm for selecting instances, labelers (and their corresponding costs and labeling accuracies), that employs generalization bound of learning with label noise to select informative instances and labelers so as to achieve higher generalization accuracy at a lower cost. Our proposed algorithm demonstrates state-of-the-art performance on five UCI and a real crowdsourcing dataset.