Inverse Propensity Score based offline estimator for deterministic ranking lists using position bias
This work addresses the challenge of evaluating ranking policies offline for practitioners in recommendation systems, but it is incremental as it builds on existing IPS methods by adding position bias modeling.
The paper tackled the problem of offline policy evaluation (OPE) for deterministic ranking lists by introducing an inverse propensity score (IPS) estimator that incorporates position bias, enabling its application to a wider range of logging policies. The result showed strong correlation with online results, though with some constant bias, validated on industry-scale data.
In this work, we present a novel way of computing IPS using a position-bias model for deterministic logging policies. This technique significantly widens the policies on which OPE can be used. We validate this technique using two different experiments on industry-scale data. The OPE results are clearly strongly correlated with the online results, with some constant bias. The estimator requires the examination model to be a reasonably accurate approximation of real user behaviour.