Evaluating LLMs as Human Surrogates in Controlled Experiments
This addresses the problem for researchers using LLMs as human surrogates in controlled experiments, highlighting limitations in their reliability.
The study evaluated whether off-the-shelf large language models (LLMs) can simulate human responses in behavioral experiments, finding that while LLMs reproduced directional effects from a survey on accuracy perception, they did not consistently match human effect magnitudes or moderation patterns.
Large language models (LLMs) are increasingly used to simulate human responses in behavioral research, yet it remains unclear when LLM-generated data support the same experimental inferences as human data. We evaluate this by directly comparing off-the-shelf LLM-generated responses with human responses from a canonical survey experiment on accuracy perception. Each human observation is converted into a structured prompt, and models generate a single 0--10 outcome variable without task-specific training; identical statistical analyses are applied to human and synthetic responses. We find that LLMs reproduce several directional effects observed in humans, but effect magnitudes and moderation patterns vary across models. Off-the-shelf LLMs therefore capture aggregate belief-updating patterns under controlled conditions but do not consistently match human-scale effects, clarifying when LLM-generated data can function as behavioral surrogates.