Designing and Deploying Online Field Experiments
This work addresses the problem for researchers and practitioners in online platforms who need to conduct complex, generalizable experiments to inform strategic decisions, though it is incremental as it builds on existing experimental design concepts.
The authors tackled the lack of tools for designing and deploying sophisticated online field experiments by introducing PlanOut, a language that separates experimental design from application code, enabling concise descriptions of complex designs like factorial or conditional logic experiments. They demonstrated its use in two large field experiments on Facebook, showing it supports iterative and parallel experimentation with a namespaced management system.
Online experiments are widely used to compare specific design alternatives, but they can also be used to produce generalizable knowledge and inform strategic decision making. Doing so often requires sophisticated experimental designs, iterative refinement, and careful logging and analysis. Few tools exist that support these needs. We thus introduce a language for online field experiments called PlanOut. PlanOut separates experimental design from application code, allowing the experimenter to concisely describe experimental designs, whether common "A/B tests" and factorial designs, or more complex designs involving conditional logic or multiple experimental units. These latter designs are often useful for understanding causal mechanisms involved in user behaviors. We demonstrate how experiments from the literature can be implemented in PlanOut, and describe two large field experiments conducted on Facebook with PlanOut. For common scenarios in which experiments are run iteratively and in parallel, we introduce a namespaced management system that encourages sound experimental practice.