Sinan Aral

Sinan Aral, Seth G Benzell, Avinash Collis et al.

Network effects -- the utility gains from additional consumers of a good -- are widely regarded as critical to the digital economy. Yet recent theory and evidence suggest that local network effects -- the economic value created by specific social network connections -- drive value in networked online platforms. Using incentive-compatible online choice experiments with 19,923 Facebook, Instagram, LinkedIn, and X users in the United States, we provide the first large-scale empirical measurement of local network effects in the digital economy and measure heterogeneity in connection value across platforms. Platform value ranges from \$78 to \$101 per consumer per month, with 8.1-23.7% explained by local network effects. We find that 1) stronger ties are more valuable on Facebook and Instagram, while weaker ties are more valuable on LinkedIn and X; 2) work connections are most valuable on LinkedIn and least on Facebook, and job-seekers value LinkedIn significantly more and Facebook significantly less; 3) men value connections to women significantly more than to other men, particularly on Instagram, Facebook, and X, while women value connections to men and women equally across platforms; 4) consumers value connections on any platform more if they are also connected on other platforms, suggesting that platforms are complements, not substitutes; 5) white consumers disproportionately value same-race connections on Facebook while, on Instagram, connections to alters eighteen or younger are valued significantly more than any other age group -- two patterns not seen on other platforms. Each platform generates between \$53B and \$215B in annual US consumer surplus. These results suggest that social media generates significant value, that local network effects drive a substantial fraction of it, and that the sources and contours of these effects vary across platforms, consumers, and connections.

Harang Ju, Sinan Aral

To uncover how AI agents change productivity, performance, and work processes, we introduce Pairit -- an experimentation platform enabling humans and AI agents to collaborate in integrative workspaces. In a large-scale marketing experiment on the platform, 2310 participants were randomly assigned to human-human and human-AI teams. The teams exchanged 183,691 messages and created 63,656 image edits, 1,960,095 ad copy edits, and 10,375 AI-generated images while producing 11,138 ads for a large think tank. Analysis of fine-grained communication, collaboration, and workflow logs revealed that collaborating with AI agents increased communication by 63% and allowed humans to engage in 71% less direct text editing. While human-AI teams engaged in 18% more process and content communication, human-human teams engaged in 29% more social and emotional communication. Humans in human-AI teams experienced 73% greater productivity per worker and produced higher-quality ad copy, while human-human teams produced higher-quality images, suggesting AI agents require fine-tuning for multimodal workflows. Field tests of the ad campaigns accumulated ~5M ad impressions and revealed that ads with higher image quality (produced by human-human collaborations) and higher text quality (produced by human-AI collaborations) performed significantly better on click-through rates, view through rates, and cost per click metrics. Together, these results suggest that human collaboration with AI agents significantly reshapes communication patterns and work processes and increases productivity, while improving some dimensions of output quality and deteriorating others. We hope the release of the extensible Pairit platform will accelerate RCTs of human-AI collaboration across a variety of work tasks and contexts.

Paramveer Dhillon, Sinan Aral

In recent years, there has been significant interest in understanding users' online content consumption patterns. But, the unstructured, high-dimensional, and dynamic nature of such data makes extracting valuable insights challenging. Here we propose a model that combines the simplicity of matrix factorization with the flexibility of neural networks to efficiently extract nonlinear patterns from massive text data collections relevant to consumers' online consumption patterns. Our model decomposes a user's content consumption journey into nonlinear user and content factors that are used to model their dynamic interests. This natural decomposition allows us to summarize each user's content consumption journey with a dynamic probabilistic weighting over a set of underlying content attributes. The model is fast to estimate, easy to interpret and can harness external data sources as an empirical prior. These advantages make our method well suited to the challenges posed by modern datasets. We use our model to understand the dynamic news consumption interests of Boston Globe readers over five years. Thorough qualitative studies, including a crowdsourced evaluation, highlight our model's ability to accurately identify nuanced and coherent consumption patterns. These results are supported by our model's superior and robust predictive performance over several competitive baseline methods.

Jeremy Yang, Dean Eckles, Paramveer Dhillon et al.

Decision makers often want to target interventions so as to maximize an outcome that is observed only in the long-term. This typically requires delaying decisions until the outcome is observed or relying on simple short-term proxies for the long-term outcome. Here we build on the statistical surrogacy and policy learning literatures to impute the missing long-term outcomes and then approximate the optimal targeting policy on the imputed outcomes via a doubly-robust approach. We first show that conditions for the validity of average treatment effect estimation with imputed outcomes are also sufficient for valid policy evaluation and optimization; furthermore, these conditions can be somewhat relaxed for policy optimization. We apply our approach in two large-scale proactive churn management experiments at The Boston Globe by targeting optimal discounts to its digital subscribers with the aim of maximizing long-term revenue. Using the first experiment, we evaluate this approach empirically by comparing the policy learned using imputed outcomes with a policy learned on the ground-truth, long-term outcomes. The performance of these two policies is statistically indistinguishable, and we rule out large losses from relying on surrogates. Our approach also outperforms a policy learned on short-term proxies for the long-term outcome. In a second field experiment, we implement the optimal targeting policy with additional randomized exploration, which allows us to update the optimal policy for future subscribers. Over three years, our approach had a net-positive revenue impact in the range of $4-5 million compared to the status quo.

David Holtz, Benjamin Carterette, Praveen Chandar et al.

It remains unknown whether personalized recommendations increase or decrease the diversity of content people consume. We present results from a randomized field experiment on Spotify testing the effect of personalized recommendations on consumption diversity. In the experiment, both control and treatment users were given podcast recommendations, with the sole aim of increasing podcast consumption. Treatment users' recommendations were personalized based on their music listening history, whereas control users were recommended popular podcasts among users in their demographic group. We find that, on average, the treatment increased podcast streams by 28.90%. However, the treatment also decreased the average individual-level diversity of podcast streams by 11.51%, and increased the aggregate diversity of podcast streams by 5.96%, indicating that personalized recommendations have the potential to create patterns of consumption that are homogenous within and diverse across users, a pattern reflecting Balkanization. Our results provide evidence of an "engagement-diversity trade-off" when recommendations are optimized solely to drive consumption: while personalized recommendations increase user engagement, they also affect the diversity of consumed content. This shift in consumption diversity can affect user retention and lifetime value, and impact the optimal strategy for content producers. We also observe evidence that our treatment affected streams from sections of Spotify's app not directly affected by the experiment, suggesting that exposure to personalized recommendations can affect the content that users consume organically. We believe these findings highlight the need for academics and practitioners to continue investing in personalization methods that explicitly take into account the diversity of content recommended.

Madhav Kumar, Dean Eckles, Sinan Aral

Bundling, the practice of jointly selling two or more products at a discount, is a widely used strategy in industry and a well examined concept in academia. Historically, the focus has been on theoretical studies in the context of monopolistic firms and assumed product relationships, e.g., complementarity in usage. We develop a new machine-learning-driven methodology for designing bundles in a large-scale, cross-category retail setting. We leverage historical purchases and consideration sets created from clickstream data to generate dense continuous representations of products called embeddings. We then put minimal structure on these embeddings and develop heuristics for complementarity and substitutability among products. Subsequently, we use the heuristics to create multiple bundles for each product and test their performance using a field experiment with a large retailer. We combine the results from the experiment with product embeddings using a hierarchical model that maps bundle features to their purchase likelihood, as measured by the add-to-cart rate. We find that our embeddings-based heuristics are strong predictors of bundle success, robust across product categories, and generalize well to the retailer's entire assortment.