CVTT: Cross-Validation Through Time
This addresses the evaluation problem for recommender systems researchers and practitioners, offering a more comprehensive method, though it is incremental as it builds on existing evaluation discourse.
The study tackled the problem of evaluating recommender systems by challenging the assumption of constant performance over time, proposing the Cross-Validation Through Time (CVTT) technique, and found that performance varies over time, simple evaluations can substantially decrease real-world performance, and excessive data usage leads to suboptimal results.
The evaluation of recommender systems from a practical perspective is a topic of ongoing discourse within the research community. While many current evaluation methods reduce performance to a single value metric as an easy way to compare models, it relies on the assumption that the methods' performance remains constant over time. In this study, we examine this assumption and propose the Cross-Validation Thought Time (CVTT) technique as a more comprehensive evaluation method, focusing on model performance over time. By utilizing the proposed technique, we conduct an in-depth analysis of the performance of popular RecSys algorithms. Our findings indicate that (1) the performance of the recommenders varies over time for all reviewed datasets, (2) using simple evaluation approaches can lead to a substantial decrease in performance in real-world evaluation scenarios, and (3) excessive data usage can lead to suboptimal results.