A Conceptual Framework for Evaluating Fairness in Search
This addresses fairness evaluation in search systems, which is an incremental contribution to the field of information retrieval.
The paper tackles the problem of evaluating fairness in search systems by defining distributional fairness and creating a conceptual framework with axioms for ideal evaluation. The result includes showing metric divergence between relevance and fairness and proposing an interpolation strategy to integrate both into a single metric.
While search efficacy has been evaluated traditionally on the basis of result relevance, fairness of search has attracted recent attention. In this work, we define a notion of distributional fairness and provide a conceptual framework for evaluating search results based on it. As part of this, we formulate a set of axioms which an ideal evaluation framework should satisfy for distributional fairness. We show how existing TREC test collections can be repurposed to study fairness, and we measure potential data bias to inform test collection design for fair search. A set of analyses show metric divergence between relevance and fairness, and we describe a simple but flexible interpolation strategy for integrating relevance and fairness into a single metric for optimization and evaluation.