The Value of Personalized Recommendations: Evidence from Netflix
This work addresses the problem of measuring recommendation system impact for platforms like Netflix, providing incremental insights into engagement and diversity effects.
The authors tackled the challenge of quantifying the value of personalized recommendation systems by developing a discrete choice model applied to Netflix viewership data, finding that replacing the current system with matrix factorization or popularity-based algorithms would reduce engagement by 4% and 12%, respectively, and that most gains come from effective targeting of mid-popularity goods.
Personalized recommendation systems shape much of user choice online, yet their targeted nature makes separating out the value of recommendation and the underlying goods challenging. We build a discrete choice model that embeds recommendation-induced utility, low-rank heterogeneity, and flexible state dependence and apply the model to viewership data at Netflix. We exploit idiosyncratic variation introduced by the recommendation algorithm to identify and separately value these components as well as to recover model-free diversion ratios that we can use to validate our structural model. We use the model to evaluate counterfactuals that quantify the incremental engagement generated by personalized recommendations. First, we show that replacing the current recommender system with a matrix factorization or popularity-based algorithm would lead to 4% and 12% reduction in engagement, respectively, and decreased consumption diversity. Second, most of the consumption increase from recommendations comes from effective targeting, not mechanical exposure, with the largest gains for mid-popularity goods (as opposed to broadly appealing or very niche goods).