How robust is MovieLens? A dataset analysis for recommender systems
This addresses reproducibility issues for researchers in recommender systems, though it is incremental as it focuses on improving existing practices rather than introducing new algorithms.
The paper tackles the problem of inconsistent preprocessing and evaluation protocols in recommender system datasets, which complicates cross-paper comparisons, and proposes a method to select protocols and enhance transparency in sharing results.
Research publication requires public datasets. In recommender systems, some datasets are largely used to compare algorithms against a --supposedly-- common benchmark. Problem: for various reasons, these datasets are heavily preprocessed, making the comparison of results across papers difficult. This paper makes explicit the variety of preprocessing and evaluation protocols to test the robustness of a dataset (or lack of flexibility). While robustness is good to compare results across papers, for flexible datasets we propose a method to select a preprocessing protocol and share results more transparently.