Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments
This work addresses the design of switchback experiments in A/B testing for industries, offering practical guidelines, but it is incremental as it builds on existing RL estimators.
The paper analyzes how carryover effects and reward autocorrelations influence the effectiveness of various switchback designs in A/B testing, revealing that these factors are crucial and estimator-agnostic, and provides a workflow for practitioners.
A/B testing has become the gold standard for policy evaluation in modern technological industries. Motivated by the widespread use of switchback experiments in A/B testing, this paper conducts a comprehensive comparative analysis of various switchback designs in Markovian environments. Unlike many existing works which derive the optimal design based on specific and relatively simple estimators, our analysis covers a range of state-of-the-art estimators developed in the reinforcement learning (RL) literature. It reveals that the effectiveness of different switchback designs depends crucially on (i) the size of the carryover effect and (ii) the auto-correlations among reward errors over time. Meanwhile, these findings are estimator-agnostic, i.e., they apply to most RL estimators. Based on these insights, we provide a workflow to offer guidelines for practitioners on designing switchback experiments in A/B testing.