Crowdsourcing with Fairness, Diversity and Budget Constraints
This addresses the issue of discriminatory labels in crowdsourced data, which is crucial for training fair machine learning algorithms, representing a novel method for a known bottleneck.
The paper tackles the problem of collecting fair and diverse data from crowdworkers under budget constraints, proposing a novel algorithm that maximizes expected accuracy while ensuring fairness in errors, with experimental validation on a real dataset showing good performance.
Recent studies have shown that the labels collected from crowdworkers can be discriminatory with respect to sensitive attributes such as gender and race. This raises questions about the suitability of using crowdsourced data for further use, such as for training machine learning algorithms. In this work, we address the problem of fair and diverse data collection from a crowd under budget constraints. We propose a novel algorithm which maximizes the expected accuracy of the collected data, while ensuring that the errors satisfy desired notions of fairness. We provide guarantees on the performance of our algorithm and show that the algorithm performs well in practice through experiments on a real dataset.