Off-policy evaluation for learning-to-rank via interpolating the item-position model and the position-based model
This work addresses the need for reliable offline evaluation in industrial recommender systems, though it is incremental as it builds on existing models.
The paper tackled the problem of offline evaluation for ranking policies in recommender systems by developing a new estimator called INTERPOL, which reduces bias compared to the position-based model and offers a better bias-variance trade-off than the item-position model, with empirical results demonstrating its effectiveness.
A critical need for industrial recommender systems is the ability to evaluate recommendation policies offline, before deploying them to production. Unfortunately, widely used off-policy evaluation methods either make strong assumptions about how users behave that can lead to excessive bias, or they make fewer assumptions and suffer from large variance. We tackle this problem by developing a new estimator that mitigates the problems of the two most popular off-policy estimators for rankings, namely the position-based model and the item-position model. In particular, the new estimator, called INTERPOL, addresses the bias of a potentially misspecified position-based model, while providing an adaptable bias-variance trade-off compared to the item-position model. We provide theoretical arguments as well as empirical results that highlight the performance of our novel estimation approach.