Evaluating Stochastic Rankings with Expected Exposure
This work addresses the need for fair and diverse ranking evaluation in information retrieval, though it is incremental as it builds on existing user models and metrics.
The paper tackles the problem of evaluating stochastic rankings by introducing expected exposure as a metric for average user attention, advocating for equal expected exposure across items of the same relevance to support objectives like fair ranking. It proposes a general evaluation methodology that relaxes classic assumptions, allowing distributions over rankings, and demonstrates its behavior across retrieval and recommendation scenarios.
We introduce the concept of \emph{expected exposure} as the average attention ranked items receive from users over repeated samples of the same query. Furthermore, we advocate for the adoption of the principle of equal expected exposure: given a fixed information need, no item should receive more or less expected exposure than any other item of the same relevance grade. We argue that this principle is desirable for many retrieval objectives and scenarios, including topical diversity and fair ranking. Leveraging user models from existing retrieval metrics, we propose a general evaluation methodology based on expected exposure and draw connections to related metrics in information retrieval evaluation. Importantly, this methodology relaxes classic information retrieval assumptions, allowing a system, in response to a query, to produce a \emph{distribution over rankings} instead of a single fixed ranking. We study the behavior of the expected exposure metric and stochastic rankers across a variety of information access conditions, including \emph{ad hoc} retrieval and recommendation. We believe that measuring and optimizing expected exposure metrics using randomization opens a new area for retrieval algorithm development and progress.