Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm
This addresses bias in search ranking for commercial engines, but is incremental as it builds on existing unbiased learning-to-rank techniques.
The paper tackles the problem of position bias in click data for learning-to-rank by proposing a novel pairwise algorithm that jointly estimates biases and trains an unbiased ranker. Experiments show it significantly outperforms existing algorithms and enhances relevance ranking in online A/B testing.
Although click data is widely used in search systems in practice, so far the inherent bias, most notably position bias, has prevented it from being used in training of a ranker for search, i.e., learning-to-rank. Recently, a number of authors have proposed new techniques referred to as 'unbiased learning-to-rank', which can reduce position bias and train a relatively high-performance ranker using click data. Most of the algorithms, based on the inverse propensity weighting (IPW) principle, first estimate the click bias at each position, and then train an unbiased ranker with the estimated biases using a learning-to-rank algorithm. However, there has not been a method for pairwise learning-to-rank that can jointly conduct debiasing of click data and training of a ranker using a pairwise loss function. In this paper, we propose a novel algorithm, which can jointly estimate the biases at click positions and the biases at unclick positions, and learn an unbiased ranker. Experiments on benchmark data show that our algorithm can significantly outperform existing algorithms. In addition, an online A/B Testing at a commercial search engine shows that our algorithm can effectively conduct debiasing of click data and enhance relevance ranking.