Inference for Batched Adaptive Experiments
This addresses a methodological problem for researchers and practitioners in economics and other fields using adaptive experiments, but it is incremental as it builds on existing inference methods.
The paper tackles the challenge of causal inference in adaptive experiments by proposing a BOLS test statistic for treatment effect inference, which aggregates per-period differences under heteroskedasticity and provides asymptotically valid confidence intervals, with simulation results showing rejection rates in scenarios with few treatment periods and varying batch sizes.
The advantages of adaptive experiments have led to their rapid adoption in economics, other fields, as well as among practitioners. However, adaptive experiments pose challenges for causal inference. This note suggests a BOLS (batched ordinary least squares) test statistic for inference of treatment effects in adaptive experiments. The statistic provides a precision-equalizing aggregation of per-period treatment-control differences under heteroskedasticity. The combined test statistic is a normalized average of heteroskedastic per-period z-statistics and can be used to construct asymptotically valid confidence intervals. We provide simulation results comparing rejection rates in the typical case with few treatment periods and few (or many) observations per batch.