SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors
This work addresses the need for robust, reproducible evaluation in social and behavioral sciences to accelerate the development of faithful LLM simulators, though it is incremental in providing a benchmark rather than a new simulation method.
The authors tackled the problem of evaluating large language models' ability to simulate human behaviors by introducing SimBench, a standardized benchmark unifying 20 diverse datasets, and found that current LLMs have limited simulation ability (score: 40.80/100), with performance scaling log-linearly with model size and showing an alignment-simulation trade-off.
Large language model (LLM) simulations of human behavior have the potential to revolutionize the social and behavioral sciences, if and only if they faithfully reflect real human behaviors. Current evaluations are fragmented, based on bespoke tasks and metrics, creating a patchwork of incomparable results. To address this, we introduce SimBench, the first large-scale, standardized benchmark for a robust, reproducible science of LLM simulation. By unifying 20 diverse datasets covering tasks from moral decision-making to economic choice across a large global participant pool, SimBench provides the necessary foundation to ask fundamental questions about when, how, and why LLM simulations succeed or fail. We show that, while even the best LLMs today have limited simulation ability (score: 40.80/100), performance scales log-linearly with model size. Simulation performance is not improved by increased inference-time compute. We demonstrate an alignment-simulation trade-off: instruction-tuning improves performance on low-entropy (consensus) questions but degrades it on high-entropy (diverse) ones. Models particularly struggle when simulating specific demographic groups. Finally, we demonstrate that simulation ability correlates most strongly with deep, knowledge-intensive reasoning (MMLU-Pro, r=0.939). By making progress measurable, we aim to accelerate the development of more faithful LLM simulators.