Consistent Position Bias Estimation without Online Interventions for Learning-to-Rank
This addresses a key challenge in search engines by enabling more reliable learning from implicit feedback, though it appears incremental as it builds on existing counterfactual learning-to-rank frameworks.
The paper tackles the problem of presentation bias in learning-to-rank systems by proposing a method for consistently estimating observation propensities without requiring manual relevance judgments, disruptive interventions, or restrictive assumptions, using only implicit feedback from multiple ranking functions. Initial simulations show the approach is scalable, accurate, and robust.
Presentation bias is one of the key challenges when learning from implicit feedback in search engines, as it confounds the relevance signal with uninformative signals due to position in the ranking, saliency, and other presentation factors. While it was recently shown how counterfactual learning-to-rank (LTR) approaches \cite{Joachims/etal/17a} can provably overcome presentation bias if observation propensities are known, it remains to show how to accurately estimate these propensities. In this paper, we propose the first method for producing consistent propensity estimates without manual relevance judgments, disruptive interventions, or restrictive relevance modeling assumptions. We merely require that we have implicit feedback data from multiple different ranking functions. Furthermore, we argue that our estimation technique applies to an extended class of Contextual Position-Based Propensity Models, where propensities not only depend on position but also on observable features of the query and document. Initial simulation studies confirm that the approach is scalable, accurate, and robust.