Adaptive Experimental Design and Counterfactual Inference
This work provides practical insights for industry practitioners dealing with non-stationarity in adaptive experimentation, though it appears incremental.
The paper addresses the challenges of using adaptive experimental design in non-stationary industrial settings, developing a framework for counterfactual inference and testing it in a commercial environment.
Adaptive experimental design methods are increasingly being used in industry as a tool to boost testing throughput or reduce experimentation cost relative to traditional A/B/N testing methods. This paper shares lessons learned regarding the challenges and pitfalls of naively using adaptive experimentation systems in industrial settings where non-stationarity is prevalent, while also providing perspectives on the proper objectives and system specifications in these settings. We developed an adaptive experimental design framework for counterfactual inference based on these experiences, and tested it in a commercial environment.