DATELINE: Deep Plackett-Luce Model with Uncertainty Measurements
This work addresses aggregation of preferences for applications like peer grading and elections, but appears incremental as it builds on existing Plackett-Luce variants.
The paper tackles the problem of aggregating k-ary preferences by addressing two issues: ignoring feature information and uncertainty in crowdsourced data, proposing DATELINE with deep neural networks and a weighted model, and provides theoretical guarantees for robustness.
The aggregation of k-ary preferences is a historical and important problem, since it has many real-world applications, such as peer grading, presidential elections and restaurant ranking. Meanwhile, variants of Plackett-Luce model has been applied to aggregate k-ary preferences. However, there are two urgent issues still existing in the current variants. First, most of them ignore feature information. Namely, they consider k-ary preferences instead of instance-dependent k-ary preferences. Second, these variants barely consider the uncertainty in k-ary preferences provided by agnostic crowds. In this paper, we propose Deep plAckeTt-luce modEL wIth uNcertainty mEasurements (DATELINE), which can address both issues simultaneously. To address the first issue, we employ deep neural networks mapping each instance into its ranking score in Plackett-Luce model. Then, we present a weighted Plackett-Luce model to solve the second issue, where the weight is a dynamic uncertainty vector measuring the worker quality. More importantly, we provide theoretical guarantees for DATELINE to justify its robustness.