Estimating Error and Bias in Offline Evaluation Results
This is an incremental study that highlights a critical limitation for researchers and developers in assessing recommendation quality, potentially hindering the evaluation of substantial improvements.
The paper tackles the problem of offline evaluation in recommender systems being inaccurate for novel recommendations due to missing data in historical datasets, finding that this leads to systematic errors in metrics and can falsely favor popularity-based recommenders over perfect personalized ones.
Offline evaluations of recommender systems attempt to estimate users' satisfaction with recommendations using static data from prior user interactions. These evaluations provide researchers and developers with first approximations of the likely performance of a new system and help weed out bad ideas before presenting them to users. However, offline evaluation cannot accurately assess novel, relevant recommendations, because the most novel items were previously unknown to the user, so they are missing from the historical data and cannot be judged as relevant. We present a simulation study to estimate the error that such missing data causes in commonly-used evaluation metrics in order to assess its prevalence and impact. We find that missing data in the rating or observation process causes the evaluation protocol to systematically mis-estimate metric values, and in some cases erroneously determine that a popularity-based recommender outperforms even a perfect personalized recommender. Substantial breakthroughs in recommendation quality, therefore, will be difficult to assess with existing offline techniques.