Online Learning for Non-Stationary A/B Tests
This addresses the problem of slow and human-intensive feature rollout in applications, offering a practical solution for developers and companies, though it appears incremental as it builds on expert learning methods.
The paper tackles the inefficiency and suboptimality of manual multi-stage A/B testing by formulating it as an expert learning problem and proposing the Follow-The-Best-Interval (FTBI) algorithm for non-stationary environments, showing that it outperforms current state-of-the-art methods in evaluations on synthetic and real-world datasets.
The rollout of new versions of a feature in modern applications is a manual multi-stage process, as the feature is released to ever larger groups of users, while its performance is carefully monitored. This kind of A/B testing is ubiquitous, but suboptimal, as the monitoring requires heavy human intervention, is not guaranteed to capture consistent, but short-term fluctuations in performance, and is inefficient, as better versions take a long time to reach the full population. In this work we formulate this question as that of expert learning, and give a new algorithm Follow-The-Best-Interval, FTBI, that works in dynamic, non-stationary environments. Our approach is practical, simple, and efficient, and has rigorous guarantees on its performance. Finally, we perform a thorough evaluation on synthetic and real world datasets and show that our approach outperforms current state-of-the-art methods.