Recommendation Systems with Distribution-Free Reliability Guarantees
This work addresses the need for reliable and interpretable recommendations in AI systems, offering a principled approach to integrate multiple objectives like diversity without ad hoc tuning, though it is incremental in applying statistical guarantees to ranking models.
The paper tackles the problem of ensuring reliability in recommendation systems by developing a method that provides rigorous finite-sample control of the false discovery rate (FDR) for any ranking model, regardless of data distribution, and demonstrates its application in optimizing for diversity while maintaining user-specified FDR levels.
When building recommendation systems, we seek to output a helpful set of items to the user. Under the hood, a ranking model predicts which of two candidate items is better, and we must distill these pairwise comparisons into the user-facing output. However, a learned ranking model is never perfect, so taking its predictions at face value gives no guarantee that the user-facing output is reliable. Building from a pre-trained ranking model, we show how to return a set of items that is rigorously guaranteed to contain mostly good items. Our procedure endows any ranking model with rigorous finite-sample control of the false discovery rate (FDR), regardless of the (unknown) data distribution. Moreover, our calibration algorithm enables the easy and principled integration of multiple objectives in recommender systems. As an example, we show how to optimize for recommendation diversity subject to a user-specified level of FDR control, circumventing the need to specify ad hoc weights of a diversity loss against an accuracy loss. Throughout, we focus on the problem of learning to rank a set of possible recommendations, evaluating our methods on the Yahoo! Learning to Rank and MSMarco datasets.