Beyond Basic A/B testing: Improving Statistical Efficiency for Business Growth
This work addresses the problem of inefficient A/B testing for businesses, offering incremental improvements to handle small samples, non-Gaussian distributions, and ROI considerations.
The paper tackles the low statistical power of standard A/B testing methods in business settings by proposing new approaches, including a novel doubly robust generalized U framework, and demonstrates efficiency gains through theoretical analysis, simulations, and real-world applications.
The standard A/B testing approaches are mostly based on t-test in large scale industry applications. These standard approaches however suffers from low statistical power in business settings, due to nature of small sample-size or non-Gaussian distribution or return-on-investment (ROI) consideration. In this paper, we propose several approaches to addresses these challenges: (i) regression adjustment, generalized estimating equation, Man-Whitney U and Zero-Trimmed U that addresses each of these issues separately, and (ii) a novel doubly robust generalized U that handles ROI consideration, distribution robustness and small samples in one framework. We provide theoretical results on asymptotic normality and efficiency bounds, together with insights on the efficiency gain from theoretical analysis. We further conduct comprehensive simulation studies and apply the methods to multiple real A/B tests.