Large Language Models as Conversational Movie Recommenders: A User Study
This work addresses the problem of conversational recommendation systems for movie viewers, but it is incremental as it evaluates existing LLMs in a new application context.
The study investigated using large language models (LLMs) for personalized movie recommendations through a user experiment with 160 participants, finding that LLMs offer strong explainability but lack personalization, diversity, and user trust, with the number of movies watched being more influential than prompting techniques.
This paper explores the effectiveness of using large language models (LLMs) for personalized movie recommendations from users' perspectives in an online field experiment. Our study involves a combination of between-subject prompt and historic consumption assessments, along with within-subject recommendation scenario evaluations. By examining conversation and survey response data from 160 active users, we find that LLMs offer strong recommendation explainability but lack overall personalization, diversity, and user trust. Our results also indicate that different personalized prompting techniques do not significantly affect user-perceived recommendation quality, but the number of movies a user has watched plays a more significant role. Furthermore, LLMs show a greater ability to recommend lesser-known or niche movies. Through qualitative analysis, we identify key conversational patterns linked to positive and negative user interaction experiences and conclude that providing personal context and examples is crucial for obtaining high-quality recommendations from LLMs.