Efficient Heterogeneous Treatment Effect Estimation With Multiple Experiments and Multiple Outcomes
This addresses the need for more efficient HTE estimation in real-world scenarios like A/B testing, where multiple metrics and experiments are common, offering a scalable solution for internet companies and other fields.
The paper tackles the problem of estimating heterogeneous treatment effects (HTEs) in settings with multiple experiments and outcomes, showing that analyzing all data together improves precision by leveraging cross-experiment and cross-outcome correlations. They propose the LR-learner model, which achieves much greater precision than independent estimation in synthetic and real data.
Learning heterogeneous treatment effects (HTEs) is an important problem across many fields. Most existing methods consider the setting with a single treatment arm and a single outcome metric. However, in many real world domains, experiments are run consistently - for example, in internet companies, A/B tests are run every day to measure the impacts of potential changes across many different metrics of interest. We show that even if an analyst cares only about the HTEs in one experiment for one metric, precision can be improved greatly by analyzing all of the data together to take advantage of cross-experiment and cross-outcome metric correlations. We formalize this idea in a tensor factorization framework and propose a simple and scalable model which we refer to as the low rank or LR-learner. Experiments in both synthetic and real data suggest that the LR-learner can be much more precise than independent HTE estimation.