Fair Effect Attribution in Parallel Online Experiments
This addresses the challenge for online platforms and experimenters in managing multiple concurrent experiments to protect user experience, though it is incremental as it builds on existing cost-sharing and causal methods.
The paper tackles the problem of negative interactions between simultaneous A/B tests on online platforms, which can harm overall user engagement metrics, by proposing a method to measure and fairly attribute these interaction effects using Shapley values and causal inference techniques, as demonstrated in real-world and synthetic experiments.
A/B tests serve the purpose of reliably identifying the effect of changes introduced in online services. It is common for online platforms to run a large number of simultaneous experiments by splitting incoming user traffic randomly in treatment and control groups. Despite a perfect randomization between different groups, simultaneous experiments can interact with each other and create a negative impact on average population outcomes such as engagement metrics. These are measured globally and monitored to protect overall user experience. Therefore, it is crucial to measure these interaction effects and attribute their overall impact in a fair way to the respective experimenters. We suggest an approach to measure and disentangle the effect of simultaneous experiments by providing a cost sharing approach based on Shapley values. We also provide a counterfactual perspective, that predicts shared impact based on conditional average treatment effects making use of causal inference techniques. We illustrate our approach in real world and synthetic data experiments.