User Validation of Recommendation Serendipity Metrics
This work addresses the gap between offline measurement and user evaluation for recommendation serendipity, which is incremental as it validates and compares existing metrics rather than introducing new ones.
The study tackled the problem of measuring serendipity in recommendations by validating existing metrics with over 10,000 users' feedback, finding that user profile-based metrics and full metrics (including unexpectedness, relevance, timeliness, and curiosity) performed better in estimating serendipity.
Though it has been recognized that recommending serendipitous (i.e., surprising and relevant) items can be helpful for increasing users' satisfaction and behavioral intention, how to measure serendipity in the offline environment is still an open issue. In recent years, a number of metrics have been proposed, but most of them were based on researchers' assumptions due to the serendipity's subjective nature. In order to validate these metrics' actual performance, we collected over 10,000 users' real feedback data and compared with the metrics' results. It turns out the user profile based metrics, especially content-based ones, perform better than those based on item popularity, in terms of estimating the unexpectedness facet of recommendations. Moreover, the full metrics, which involve the unexpectedness component, relevance, timeliness, and user curiosity, can more accurately indicate the recommendation's serendipity degree, relative to those that just involve some of them. The application of these metrics to several recommender algorithms further consolidates their practical usage, because the comparison results are consistent with those from user evaluation. Thus, this work is constructive for filling the gap between offline measurement and user study on recommendation serendipity.