Offline A/B testing for Recommender Systems
This work addresses the need for faster iteration and cost reduction in deploying new recommender technologies for online advertising, though it is incremental as it builds on existing counterfactual methods.
The paper tackled the problem of offline A/B testing for recommender systems by proposing two new counterfactual estimators that model bias differently, showing they are accurate in real-world conditions and correlate well with online business metrics.
Before A/B testing online a new version of a recommender system, it is usual to perform some offline evaluations on historical data. We focus on evaluation methods that compute an estimator of the potential uplift in revenue that could generate this new technology. It helps to iterate faster and to avoid losing money by detecting poor policies. These estimators are known as counterfactual or off-policy estimators. We show that traditional counterfactual estimators such as capped importance sampling and normalised importance sampling are experimentally not having satisfying bias-variance compromises in the context of personalised product recommendation for online advertising. We propose two variants of counterfactual estimates with different modelling of the bias that prove to be accurate in real-world conditions. We provide a benchmark of these estimators by showing their correlation with business metrics observed by running online A/B tests on a commercial recommender system.