Position bias in features
This work addresses the challenge of improving search relevance for users by highlighting the trade-offs in using unbiased features, but it is incremental as it builds on existing methods like inverse propensity weighting.
The paper tackles the problem of position bias in search engine ranking features, showing that an unbiased feature based on inverse propensity weighting can achieve near-optimal ranking but suffers from high variance and poor performance if position bias estimation is inaccurate, sometimes underperforming biased click-through rates.
The purpose of modeling document relevance for search engines is to rank better in subsequent searches. Document-specific historical click-through rates can be important features in a dynamic ranking system which updates as we accumulate more sample. This paper describes the properties of several such features, and tests them in controlled experiments. Extending the inverse propensity weighting method to documents creates an unbiased estimate of document relevance. This feature can approximate relevance accurately, leading to near-optimal ranking in ideal circumstances. However, it has high variance that is increasing with respect to the degree of position bias. Furthermore, inaccurate position bias estimation leads to poor performance. Under several scenarios this feature can perform worse than biased click-through rates. This paper underscores the need for accurate position bias estimation, and is unique in suggesting simultaneous use of biased and unbiased position bias features.