Pareto optimal proxy metrics
This addresses the challenge for technology companies in making faster and higher-quality launch decisions for product features, though it is incremental as it builds on existing proxy metric literature.
The paper tackles the problem of low sensitivity and short-term vs. long-term trade-offs in evaluating experiments using north star metrics by proposing Pareto optimal proxy metrics that optimize both prediction accuracy and sensitivity, resulting in metrics eight times more sensitive than the north star in a large industrial recommendation system.
North star metrics and online experimentation play a central role in how technology companies improve their products. In many practical settings, however, evaluating experiments based on the north star metric directly can be difficult. The two most significant issues are 1) low sensitivity of the north star metric and 2) differences between the short-term and long-term impact on the north star metric. A common solution is to rely on proxy metrics rather than the north star in experiment evaluation and launch decisions. Existing literature on proxy metrics concentrates mainly on the estimation of the long-term impact from short-term experimental data. In this paper, instead, we focus on the trade-off between the estimation of the long-term impact and the sensitivity in the short term. In particular, we propose the Pareto optimal proxy metrics method, which simultaneously optimizes prediction accuracy and sensitivity. In addition, we give an efficient multi-objective optimization algorithm that outperforms standard methods. We applied our methodology to experiments from a large industrial recommendation system, and found proxy metrics that are eight times more sensitive than the north star and consistently moved in the same direction, increasing the velocity and the quality of the decisions to launch new features.