Assessing the Reliability of Persona-Conditioned LLMs as Synthetic Survey Respondents
This highlights a key adverse impact for computational social science, showing that current persona-based simulation practices can undermine subgroup fidelity and mislead analyses.
The study assessed whether persona-conditioned LLMs improve reliability as synthetic survey respondents, finding that persona prompting does not enhance aggregate survey alignment and often degrades performance, with disproportionate distortions in underrepresented subgroups.
Using persona-conditioned LLMs as synthetic survey respondents has become a common practice in computational social science and agent-based simulations. Yet, it remains unclear whether multi-attribute persona prompting improves LLM reliability or instead introduces distortions. Here we contribute to this assessment by leveraging a large dataset of U.S. microdata from the World Values Survey. Concretely, we evaluate two open-weight chat models and a random-guesser baseline across more than 70K respondent-item instances. We find that persona prompting does not yield a clear aggregate improvement in survey alignment and, in many cases, significantly degrades performance. Persona effects are highly heterogeneous as most items exhibit minimal change, while a small subset of questions and underrepresented subgroups experience disproportionate distortions. Our findings highlight a key adverse impact of current persona-based simulation practices: demographic conditioning can redistribute error in ways that undermine subgroup fidelity and risk misleading downstream analyses.