Optimizing positional scoring rules for rank aggregation
This work addresses a fundamental optimization problem in social choice for crowdsourcing systems, but it is incremental as it builds on existing positional scoring rules with theoretical and experimental extensions.
The paper tackles the problem of selecting the best positional scoring rule for rank aggregation in crowdsourcing by optimizing the rule to match a partially known true ranking, and presents theoretical complexity results and experimental validation on real-world and synthetic data.
Nowadays, several crowdsourcing projects exploit social choice methods for computing an aggregate ranking of alternatives given individual rankings provided by workers. Motivated by such systems, we consider a setting where each worker is asked to rank a fixed (small) number of alternatives and, then, a positional scoring rule is used to compute the aggregate ranking. Among the apparently infinite such rules, what is the best one to use? To answer this question, we assume that we have partial access to an underlying true ranking. Then, the important optimization problem to be solved is to compute the positional scoring rule whose outcome, when applied to the profile of individual rankings, is as close as possible to the part of the underlying true ranking we know. We study this fundamental problem from a theoretical viewpoint and present positive and negative complexity results and, furthermore, complement our theoretical findings with experiments on real-world and synthetic data.