Mediation Analysis in Online Experiments at Booking.com: Disentangling Direct and Indirect Effects
This work provides a practical framework for mediation analysis in online experimentation, helping product teams at companies like Booking.com make better decisions by understanding causal mechanisms, though it is incremental in applying existing statistical methods to this domain.
The paper addresses the problem of distinguishing direct and indirect effects in online experiments at Booking.com, demonstrating through simulated data how mediation analysis helps identify direct causal effects and providing sensitivity analyses to assess robustness to missing confounders.
Online experimentation is at the core of Booking.com's customer-centric product development. While randomised controlled trials are a powerful tool for estimating the overall effects of product changes on business metrics, they often fall short in explaining the mechanism of change. This becomes problematic when decision-making depends on being able to distinguish between the direct effect of a treatment on some outcome variable and its indirect effect via a mediator variable. In this paper, we demonstrate the need for mediation analyses in online experimentation, and use simulated data to show how these methods help identify and estimate direct causal effect. Failing to take into account all confounders can lead to biased estimates, so we include sensitivity analyses to help gauge the robustness of estimates to missing causal factors.