Do recommender systems benefit users?
This work addresses the problem of evaluating real user benefit in recommender systems for operators and researchers, highlighting that high accuracy metrics may not translate to actual benefit.
The paper investigates whether recommender systems benefit users, finding that recommendations can be equivalent to random draws if users rely too strongly on the system, but with sufficient user preference information, an abrupt transition to accurate recommendations occurs for some algorithms.
Recommender systems are present in many web applications to guide our choices. They increase sales and benefit sellers, but whether they benefit customers by providing relevant products is questionable. Here we introduce a model to examine the benefit of recommender systems for users, and found that recommendations from the system can be equivalent to random draws if one relies too strongly on the system. Nevertheless, with sufficient information about user preferences, recommendations become accurate and an abrupt transition to this accurate regime is observed for some algorithms. On the other hand, we found that a high accuracy evaluated by common accuracy metrics does not necessarily correspond to a high real accuracy nor a benefit for users, which serves as an alarm for operators and researchers of recommender systems. We tested our model with a real dataset and observed similar behaviors. Finally, a recommendation approach with improved accuracy is suggested. These results imply that recommender systems can benefit users, but relying too strongly on the system may render the system ineffective.