An Improved Multileaving Algorithm for Online Ranker Evaluation
This addresses a key challenge in information retrieval for improving the efficiency and accuracy of online ranker evaluation, though it is incremental as it builds on prior multileaving methods.
The paper tackles the problem of existing multileaving methods in online ranker evaluation not scaling well or producing results that differ from measures like NDCG due to unaccounted similarities between rankers, and proposes a new method that reduces errors by up to 50% in some cases.
Online ranker evaluation is a key challenge in information retrieval. An important task in the online evaluation of rankers is using implicit user feedback for inferring preferences between rankers. Interleaving methods have been found to be efficient and sensitive, i.e. they can quickly detect even small differences in quality. It has recently been shown that multileaving methods exhibit similar sensitivity but can be more efficient than interleaving methods. This paper presents empirical results demonstrating that existing multileaving methods either do not scale well with the number of rankers, or, more problematically, can produce results which substantially differ from evaluation measures like NDCG. The latter problem is caused by the fact that they do not correctly account for the similarities that can occur between rankers being multileaved. We propose a new multileaving method for handling this problem and demonstrate that it substantially outperforms existing methods, in some cases reducing errors by as much as 50%.