Greedy Optimized Multileaving for Personalization
This addresses the need for faster and more efficient online evaluation of personalized rankings in services like news recommenders, though it appears incremental as it builds on existing multileaving techniques.
The paper tackled the problem of efficiently evaluating personalized rankings in recommender systems by optimizing multileaving methods, resulting in Greedy Optimized Multileaving (GOM) that achieved precise evaluation with sample sizes less than 1/10 of A/B testing.
Personalization plays an important role in many services. To evaluate personalized rankings, online evaluation, such as A/B testing, is widely used today. Recently, multileaving has been found to be an efficient method for evaluating rankings in information retrieval fields. This paper describes the first attempt to optimize the multileaving method for personalization settings. We clarify the challenges of applying this method to personalized rankings. Then, to solve these challenges, we propose greedy optimized multileaving (GOM) with a new credit feedback function. The empirical results showed that GOM was stable for increasing ranking lengths and the number of rankers. We implemented GOM on our actual news recommender systems, and compared its online performance. The results showed that GOM evaluated the personalized rankings precisely, with significantly smaller sample sizes (< 1/10) than A/B testing.