Estimating Treatment Effects under Recommender Interference: A Structured Neural Networks Approach
This addresses a critical issue for content-sharing platforms that rely on randomized experiments to make business decisions, by providing a method to correct for interference bias, though it is incremental as it builds on existing double machine learning frameworks.
The paper tackles the problem of biased treatment effect estimates in recommender system experiments due to recommender interference, and shows that their proposed debiased estimator successfully recovers interference-free ground truth in a large-scale field experiment, while benchmark estimators exhibit substantial bias or reversed signs.
Recommender systems are essential for content-sharing platforms by curating personalized content. To improve recommender systems, platforms frequently rely on creator-side randomized experiments to evaluate algorithm updates. We show that commonly adopted difference-in-means estimators can lead to severely biased estimates due to recommender interference, where treated and control creators compete for exposure. This bias can result in incorrect business decisions. To address this, we propose a ``recommender choice model'' that explicitly represents the interference pathway. The approach combines a structural choice framework with neural networks to account for rich viewer-content heterogeneity. Building on this foundation, we develop a debiased estimator using the double machine learning (DML) framework to adjust for errors from nuisance component estimation. We show that the estimator is $\sqrt{n}$-consistent and asymptotically normal, and we extend the DML theory to handle correlated data, which arise in our context due to overlapped items. We validate our method with a large-scale field experiment on Weixin short-video platform, using a costly double-sided randomization design to obtain an interference-free ground truth. Our results show that the proposed estimator successfully recovers this ground truth, whereas benchmark estimators exhibit substantial bias, and in some cases, yield reversed signs.