Does It Look Sequential? An Analysis of Datasets for Evaluation of Sequential Recommendations
This work addresses the problem of ensuring valid evaluation for sequential recommender systems by identifying datasets with weak sequential structure, which is crucial for researchers in the field.
The paper analyzed 15 datasets used for evaluating sequential recommender systems by applying random shuffling to assess sequential structure, finding that several popular datasets have weak sequential patterns.
Sequential recommender systems are an important and demanded area of research. Such systems aim to use the order of interactions in a user's history to predict future interactions. The premise is that the order of interactions and sequential patterns play an essential role. Therefore, it is crucial to use datasets that exhibit a sequential structure to evaluate sequential recommenders properly. We apply several methods based on the random shuffling of the user's sequence of interactions to assess the strength of sequential structure across 15 datasets, frequently used for sequential recommender systems evaluation in recent research papers presented at top-tier conferences. As shuffling explicitly breaks sequential dependencies inherent in datasets, we estimate the strength of sequential patterns by comparing metrics for shuffled and original versions of the dataset. Our findings show that several popular datasets have a rather weak sequential structure.