SimUSER: Simulating User Behavior with Large Language Models for Recommender System Evaluation
This work addresses the problem of offline evaluation gaps for recommender system developers, though it is incremental as it builds on existing simulation methods.
The paper tackles the challenge of evaluating recommender systems by introducing SimUSER, an agent framework that simulates user behavior using large language models, which shows closer alignment to real humans and leads to improved user engagement in real-world applications.
Recommender systems play a central role in numerous real-life applications, yet evaluating their performance remains a significant challenge due to the gap between offline metrics and online behaviors. Given the scarcity and limits (e.g., privacy issues) of real user data, we introduce SimUSER, an agent framework that serves as believable and cost-effective human proxies. SimUSER first identifies self-consistent personas from historical data, enriching user profiles with unique backgrounds and personalities. Then, central to this evaluation are users equipped with persona, memory, perception, and brain modules, engaging in interactions with the recommender system. SimUSER exhibits closer alignment with genuine humans than prior work, both at micro and macro levels. Additionally, we conduct insightful experiments to explore the effects of thumbnails on click rates, the exposure effect, and the impact of reviews on user engagement. Finally, we refine recommender system parameters based on offline A/B test results, resulting in improved user engagement in the real world.