Ranking metrics on non-shuffled traffic
This work tackles the issue of accurate evaluation in recommender systems without compromising user experience, though it appears incremental as it builds on existing stochastic policy approaches.
The paper addresses the problem of position bias in ranking metrics for recommender systems, which typically require shuffling recommendations and can degrade user experience, by proposing a new method that leverages the stochasticity of the policy used to collect the dataset.
Ranking metrics are a family of metrics largely used to evaluate recommender systems. However they typically suffer from the fact the reward is affected by the order in which recommended items are displayed to the user. A classical way to overcome this position bias is to uniformly shuffle a proportion of the recommendations, but this method may result in a bad user experience. It is nevertheless common to use a stochastic policy to generate the recommendations, and we suggest a new method to overcome the position bias, by leveraging the stochasticity of the policy used to collect the dataset.