Evaluating the Use of Large Language Models as Synthetic Social Agents in Social Science Research
This addresses the risk of misinterpretation for social science researchers using LLMs, offering incremental guidance to improve reliability.
The paper tackles the problem of interpreting Large Language Models (LLMs) as synthetic agents in social science research by proposing a reframing as high-capacity pattern matchers with explicit scope conditions, and introduces practical guardrails like independent draws and preregistered human baselines to avoid category errors.
Large Language Models (LLMs) are being increasingly used as synthetic agents in social science, in applications ranging from augmenting survey responses to powering multi-agent simulations. This paper outlines cautions that should be taken when interpreting LLM outputs and proposes a pragmatic reframing for the social sciences in which LLMs are used as high-capacity pattern matchers for quasi-predictive interpolation under explicit scope conditions and not as substitutes for probabilistic inference. Practical guardrails such as independent draws, preregistered human baselines, reliability-aware validation, and subgroup calibration, are introduced so that researchers may engage in useful prototyping and forecasting while avoiding category errors.