Early Detection of Long Term Evaluation Criteria in Online Controlled Experiments
This work addresses the challenge for practitioners in online advertising and A/B testing who need to optimize for long-term metrics rather than short-term ones, though it is incremental in nature.
The paper tackles the problem of early detection of long-term effects in online experiments, presenting a bootstrap-based method for forecasting lifetime differences between groups, which is applied to online advertising to accelerate testing periods.
A common dilemma encountered by many upon implementing an optimization method or experiment, whether it be a reinforcement learning algorithm, or A/B testing, is deciding on what metric to optimize for. Very often short-term metrics, which are easier to measure are chosen over long term metrics which have undesirable time considerations and often a more complex calculation. In this paper, we argue the importance of choosing a metrics that focuses on long term effects. With this comes the necessity in the ability to measure significant differences between groups relatively early. We present here an efficient methodology for early detection of lifetime differences between groups based on bootstrap hypothesis testing of the lifetime forecast of the response. We present an application of this method in the domain of online advertising and we argue that approach not only allows one to focus on the ultimate metric of importance but also provides a means of accelerating the testing period.