Ranking by Lifts: A Cost-Benefit Approach to Large-Scale A/B Tests
This addresses the need for cost-effective decision-making in business experimentation on digital platforms, though it is incremental as it builds on existing FDR-controlling methods.
The paper tackled the problem of prioritizing A/B tests by maximizing expected profit while controlling cost-weighted false discoveries, resulting in a decision-theoretic framework with an empirical Bayes approach that demonstrated superior performance over existing methods in large-scale settings.
A/B testing is a core tool for decision-making in business experimentation, particularly in digital platforms and marketplaces. Practitioners often prioritize lift in performance metrics while seeking to control the costs of false discoveries. This paper develops a decision-theoretic framework for maximizing expected profit subject to a constraint on the cost-weighted false discovery rate (FDR). We propose an empirical Bayes approach that uses a greedy knapsack algorithm to rank experiments based on the ratio of expected lift to cost, incorporating the local false discovery rate (lfdr) as a key statistic. The resulting oracle rule is valid and rank-optimal. In large-scale settings, we establish the asymptotic validity of a data-driven implementation and demonstrate superior finite-sample performance over existing FDR-controlling methods. An application to A/B tests run on the Optimizely platform highlights the business value of the approach.