Efficient Elicitation of Collective Disagreements
For researchers in social choice and preference aggregation, the paper provides a theoretical framework and practical protocols to elicit richer disagreement information beyond pairwise comparisons.
The paper analyzes the structure of disagreement among voters, showing that pairwise comparisons are insufficient to capture certain disagreement measures (e.g., rank-variance, divisiveness) which require level-3 information (subsets of size 3). It introduces the plurality matrix and designs elicitation protocols to estimate it, balancing participant numbers and cognitive load.
We analyze the structure of the disagreement among a population of voters over a set of alternatives. Surveys typically ask either for pairwise comparisons, simple and intuitive for participants, or full rankings over alternatives, eliciting the entire voters' preferences. Building on the observation that pairwise comparisons cannot distinguish structural disagreement from noise, we propose a stratified framework to identify the minimal aggregated preference information needed to compute a number of disagreement measures from the literature. Specifically, we introduce the plurality matrix, a generalization of pairwise comparisons that records, for every subset $S$ of alternatives, the probability that each $a \in S$ ranks first in $S$. We define the level of a disagreement measure as the smallest subset size needed to express it, showing that many existing notions, including rank-variance and divisiveness, sit at level $3$, proving that pairwise comparisons are not enough. In addition, we demonstrate the interest of going beyond level $3$ both theoretically and experimentally. To make these results actionable, we design two elicitation protocols to estimate the plurality matrix, exploring the trade-off between the number of required participants and the cognitive load requested to each of them.