A Design-based Solution for Causal Inference with Text: Can a Language Model Be Too Large?
This work addresses a critical issue in social science and policy-making for estimating causal effects from text data, though it is incremental in proposing a design-based solution rather than a novel method.
The authors tackled the problem of isolating causal effects of text properties, such as linguistic features, on audience attitudes by introducing a new experimental design that avoids overlap bias from large language models. They demonstrated that LLM-based methods underperform simpler models in a real experiment on political communication, isolating the causal effect of expressing humility on persuasiveness with concrete results from their study.
Many social science questions ask how linguistic properties causally affect an audience's attitudes and behaviors. Because text properties are often interlinked (e.g., angry reviews use profane language), we must control for possible latent confounding to isolate causal effects. Recent literature proposes adapting large language models (LLMs) to learn latent representations of text that successfully predict both treatment and the outcome. However, because the treatment is a component of the text, these deep learning methods risk learning representations that actually encode the treatment itself, inducing overlap bias. Rather than depending on post-hoc adjustments, we introduce a new experimental design that handles latent confounding, avoids the overlap issue, and unbiasedly estimates treatment effects. We apply this design in an experiment evaluating the persuasiveness of expressing humility in political communication. Methodologically, we demonstrate that LLM-based methods perform worse than even simple bag-of-words models using our real text and outcomes from our experiment. Substantively, we isolate the causal effect of expressing humility on the perceived persuasiveness of political statements, offering new insights on communication effects for social media platforms, policy makers, and social scientists.