Confidence Intervals for Policy Evaluation in Adaptive Experiments
This work addresses statistical inference challenges for researchers and practitioners using adaptive experiments, such as in clinical trials or online platforms, but it is incremental as it builds on existing inverse propensity weighting methods.
The paper tackles the problem of biased and heavy-tailed estimators in adaptive experiments, particularly when evaluating sub-optimal treatments, by introducing an adaptive reweighting scheme for augmented inverse propensity weighting estimators, resulting in asymptotically correct coverage and reduced variance validated through numerical experiments with improved RMSE and coverage.
Adaptive experiment designs can dramatically improve statistical efficiency in randomized trials, but they also complicate statistical inference. For example, it is now well known that the sample mean is biased in adaptive trials. Inferential challenges are exacerbated when our parameter of interest differs from the parameter the trial was designed to target, such as when we are interested in estimating the value of a sub-optimal treatment after running a trial to determine the optimal treatment using a stochastic bandit design. In this context, typical estimators that use inverse propensity weighting to eliminate sampling bias can be problematic: their distributions become skewed and heavy-tailed as the propensity scores decay to zero. In this paper, we present a class of estimators that overcome these issues. Our approach is to adaptively reweight the terms of an augmented inverse propensity weighting estimator to control the contribution of each term to the estimator's variance. This adaptive weighting scheme prevents estimates from becoming heavy-tailed, ensuring asymptotically correct coverage. It also reduces variance, allowing us to test hypotheses with greater power - especially hypotheses that were not targeted by the experimental design. We validate the accuracy of the resulting estimates and their confidence intervals in numerical experiments and show our methods compare favorably to existing alternatives in terms of RMSE and coverage.