What's in a Query: Polarity-Aware Distribution-Based Fair Ranking
This work addresses fairness in rankings for safety-critical applications like search exposure, offering a more reliable approach to prevent unfairness over sequences of user queries.
The paper tackles the problem of ensuring fairness in machine learning-driven rankings by proposing new divergence-based measures for attention distribution fairness (DistFaiR), which characterize unfairness as the divergence between attention and relevance distributions over time. It proves that group fairness is upper-bounded by individual fairness under this definition and shows experimentally that maximizing individual fairness often benefits group fairness, while also identifying a fairwashing risk in prior methods.
Machine learning-driven rankings, where individuals (or items) are ranked in response to a query, mediate search exposure or attention in a variety of safety-critical settings. Thus, it is important to ensure that such rankings are fair. Under the goal of equal opportunity, attention allocated to an individual on a ranking interface should be proportional to their relevance across search queries. In this work, we examine amortized fair ranking -- where relevance and attention are cumulated over a sequence of user queries to make fair ranking more feasible in practice. Unlike prior methods that operate on expected amortized attention for each individual, we define new divergence-based measures for attention distribution-based fairness in ranking (DistFaiR), characterizing unfairness as the divergence between the distribution of attention and relevance corresponding to an individual over time. This allows us to propose new definitions of unfairness, which are more reliable at test time. Second, we prove that group fairness is upper-bounded by individual fairness under this definition for a useful class of divergence measures, and experimentally show that maximizing individual fairness through an integer linear programming-based optimization is often beneficial to group fairness. Lastly, we find that prior research in amortized fair ranking ignores critical information about queries, potentially leading to a fairwashing risk in practice by making rankings appear more fair than they actually are.