What Are We Optimizing For? A Human-centric Evaluation of Deep Learning-based Movie Recommenders
This work addresses the gap in human-centric evaluation for recommender systems, highlighting the need to balance multiple user experience attributes beyond accuracy, which is crucial for improving real-world applications in domains like entertainment.
The study evaluated four deep learning-based movie recommenders through a survey of 445 users, finding that while some models excelled in novelty and serendipity, they underperformed in diversity, trustworthiness, transparency, accuracy, and overall satisfaction compared to classic collaborative filtering methods.
In the past decade, deep learning (DL) models have gained prominence for their exceptional accuracy on benchmark datasets in recommender systems (RecSys). However, their evaluation has primarily relied on offline metrics, overlooking direct user perception and experience. To address this gap, we conduct a human-centric evaluation case study of four leading DL-RecSys models in the movie domain. We test how different DL-RecSys models perform in personalized recommendation generation by conducting survey study with 445 real active users. We find some DL-RecSys models to be superior in recommending novel and unexpected items and weaker in diversity, trustworthiness, transparency, accuracy, and overall user satisfaction compared to classic collaborative filtering (CF) methods. To further explain the reasons behind the underperformance, we apply a comprehensive path analysis. We discover that the lack of diversity and too much serendipity from DL models can negatively impact the consequent perceived transparency and personalization of recommendations. Such a path ultimately leads to lower summative user satisfaction. Qualitatively, we confirm with real user quotes that accuracy plus at least one other attribute is necessary to ensure a good user experience, while their demands for transparency and trust can not be neglected. Based on our findings, we discuss future human-centric DL-RecSys design and optimization strategies.