Beyond Averages: Evaluating LLMs on Human Survey Replication at the Distributional Level
For researchers using LLMs to simulate human behavior, this paper shows that distributional replication is poor and mean-based metrics can be deceptive, highlighting a critical limitation in current evaluation practices.
LLMs replicate mean-level human survey responses but fail to capture distributional variability, with no model outperforming a condition-insensitive baseline for purchase quantity, and mean-based evaluation alone being misleading.
LLMs are increasingly used to simulate human survey responses, but prior work has mainly evaluated replication using mean-level or aggregate agreement, offering limited insight into whether LLMs reproduce the variability of human behavior. We evaluate LLM-based survey replication at the distributional level using a non-public 2010 consumer choice experiment on Korean instant noodle purchases, a setting unlikely to overlap with model training data. We evaluate three response variables of differing statistical type: binary purchase incidence, categorical brand choice, and count purchase quantity. For each, we compare human and LLM responses at mean-level, pattern, and distributional alignment, and against reference baselines from the human data alone. LLMs reproduce condition-level patterns reasonably well but fail to capture distributional structure: for purchase quantity, no model beats a condition-insensitive baseline that simply matches the pooled human distribution. Because models that match human means well can still produce distributions further from humans than this baseline, mean-based evaluation alone can be actively misleading. Replication also varies with input configuration, with structured personas and multimodal inputs improving alignment while explicit reasoning prompting degrades it monotonically.