RADio -- Rank-Aware Divergence Metrics to Measure Normative Diversity in News Recommendations
This work addresses the need for better diversity metrics in news recommender systems, offering a domain-specific improvement over traditional similarity-based approaches.
The paper tackles the problem of measuring diversity in news recommendations by introducing RADio, a framework that evaluates normative diversity based on news organizations' norms and values, and finds that it provides insightful estimates for informing recommender system design.
In traditional recommender system literature, diversity is often seen as the opposite of similarity, and typically defined as the distance between identified topics, categories or word models. However, this is not expressive of the social science's interpretation of diversity, which accounts for a news organization's norms and values and which we here refer to as normative diversity. We introduce RADio, a versatile metrics framework to evaluate recommendations according to these normative goals. RADio introduces a rank-aware Jensen Shannon (JS) divergence. This combination accounts for (i) a user's decreasing propensity to observe items further down a list and (ii) full distributional shifts as opposed to point estimates. We evaluate RADio's ability to reflect five normative concepts in news recommendations on the Microsoft News Dataset and six (neural) recommendation algorithms, with the help of our metadata enrichment pipeline. We find that RADio provides insightful estimates that can potentially be used to inform news recommender system design.