A nonparametric sequential test for online randomized experiments
This addresses practical issues for researchers and practitioners in online experimentation, offering a robust method for continuous monitoring without distributional assumptions, though it is incremental as it builds on existing sequential testing and bootstrap techniques.
The paper tackled the problem of conducting hypothesis tests for complex metrics and preventing type 1 error inflation in online randomized experiments, proposing a nonparametric sequential test that controls type 1 error at any time and shows good power and robustness in validation on e-commerce data.
We propose a nonparametric sequential test that aims to address two practical problems pertinent to online randomized experiments: (i) how to do a hypothesis test for complex metrics; (ii) how to prevent type $1$ error inflation under continuous monitoring. The proposed test does not require knowledge of the underlying probability distribution generating the data. We use the bootstrap to estimate the likelihood for blocks of data followed by mixture sequential probability ratio test. We validate this procedure on data from a major online e-commerce website. We show that the proposed test controls type $1$ error at any time, has good power, is robust to misspecification in the distribution generating the data, and allows quick inference in online randomized experiments.