Correcting for Selection Bias in Learning-to-rank Systems
This work addresses a critical bias issue in recommendation systems for users and developers, but it is incremental as it builds on existing bias correction methods by focusing on a less-studied aspect.
The paper tackles the problem of selection bias in learning-to-rank systems, which arises because clicked documents only reflect what was shown to users, and proposes new counterfactual methods that also account for position bias, resulting in improved robustness to noise and better accuracy compared to existing unbiased algorithms, especially when position bias is moderate or absent.
Click data collected by modern recommendation systems are an important source of observational data that can be utilized to train learning-to-rank (LTR) systems. However, these data suffer from a number of biases that can result in poor performance for LTR systems. Recent methods for bias correction in such systems mostly focus on position bias, the fact that higher ranked results (e.g., top search engine results) are more likely to be clicked even if they are not the most relevant results given a user's query. Less attention has been paid to correcting for selection bias, which occurs because clicked documents are reflective of what documents have been shown to the user in the first place. Here, we propose new counterfactual approaches which adapt Heckman's two-stage method and accounts for selection and position bias in LTR systems. Our empirical evaluation shows that our proposed methods are much more robust to noise and have better accuracy compared to existing unbiased LTR algorithms, especially when there is moderate to no position bias.