SVP-CF: Selection via Proxy for Collaborative Filtering Data
This addresses a practical issue for recommender system developers by improving the reliability of algorithm comparisons during training and evaluation, though it is incremental as it builds on existing sampling methods.
The paper tackles the problem of dataset sampling strategies affecting recommendation algorithm performance, showing that common schemes can mask deficiencies or alter relative performance. It introduces SVP-CF, a data-specific sampling strategy that more accurately preserves the relative ranking of algorithms, especially for long-tail interaction data.
We study the practical consequences of dataset sampling strategies on the performance of recommendation algorithms. Recommender systems are generally trained and evaluated on samples of larger datasets. Samples are often taken in a naive or ad-hoc fashion: e.g. by sampling a dataset randomly or by selecting users or items with many interactions. As we demonstrate, commonly-used data sampling schemes can have significant consequences on algorithm performance -- masking performance deficiencies in algorithms or altering the relative performance of algorithms, as compared to models trained on the complete dataset. Following this observation, this paper makes the following main contributions: (1) characterizing the effect of sampling on algorithm performance, in terms of algorithm and dataset characteristics (e.g. sparsity characteristics, sequential dynamics, etc.); and (2) designing SVP-CF, which is a data-specific sampling strategy, that aims to preserve the relative performance of models after sampling, and is especially suited to long-tail interaction data. Detailed experiments show that SVP-CF is more accurate than commonly used sampling schemes in retaining the relative ranking of different recommendation algorithms.