Strategic A/B testing via Maximum Probability-driven Two-armed Bandit
This work addresses the problem of identifying small but economically impactful effects in A/B testing for large-scale applications, representing an incremental advancement in statistical methodology.
The paper tackled the challenge of detecting minor average treatment effects in large-scale A/B testing, where traditional methods often fail due to low sensitivity, and proposed a maximum probability-driven two-armed bandit process that improves statistical power, as demonstrated by experimental results showing significant improvements and potential cost reductions.
Detecting a minor average treatment effect is a major challenge in large-scale applications, where even minimal improvements can have a significant economic impact. Traditional methods, reliant on normal distribution-based or expanded statistics, often fail to identify such minor effects because of their inability to handle small discrepancies with sufficient sensitivity. This work leverages a counterfactual outcome framework and proposes a maximum probability-driven two-armed bandit (TAB) process by weighting the mean volatility statistic, which controls Type I error. The implementation of permutation methods further enhances the robustness and efficacy. The established strategic central limit theorem (SCLT) demonstrates that our approach yields a more concentrated distribution under the null hypothesis and a less concentrated one under the alternative hypothesis, greatly improving statistical power. The experimental results indicate a significant improvement in the A/B testing, highlighting the potential to reduce experimental costs while maintaining high statistical power.