On the Difficulty of Evaluating Baselines: A Study on Recommender Systems
This study reveals critical flaws in empirical research practices for recommender systems, indicating that findings may be unreliable without standardized benchmarks.
The paper demonstrates that baseline evaluations in recommender systems are often suboptimal, showing that a carefully tuned vanilla matrix factorization baseline outperforms reported results of new methods on the Movielens 10M dataset and highlighting the effort needed for high-quality results on the Netflix Prize.
Numerical evaluations with comparisons to baselines play a central role when judging research in recommender systems. In this paper, we show that running baselines properly is difficult. We demonstrate this issue on two extensively studied datasets. First, we show that results for baselines that have been used in numerous publications over the past five years for the Movielens 10M benchmark are suboptimal. With a careful setup of a vanilla matrix factorization baseline, we are not only able to improve upon the reported results for this baseline but even outperform the reported results of any newly proposed method. Secondly, we recap the tremendous effort that was required by the community to obtain high quality results for simple methods on the Netflix Prize. Our results indicate that empirical findings in research papers are questionable unless they were obtained on standardized benchmarks where baselines have been tuned extensively by the research community.