To Mask or to Mirror: Human-AI Alignment in Collective Reasoning
This work addresses the critical issue of ensuring AI systems align with human social dynamics in collective settings, which is essential for advancing socially-aligned AI, though it is incremental as it builds on prior individual-level alignment research.
The study tackled the problem of aligning large language models (LLMs) with human collective reasoning in decision-making, finding that LLM behaviors diverge—some mirror human biases while others mask them—depending on context and model-specific factors, based on a large-scale experiment with 748 participants and simulations of multiple LLMs.
As large language models (LLMs) are increasingly used to model and augment collective decision-making, it is critical to examine their alignment with human social reasoning. We present an empirical framework for assessing collective alignment, in contrast to prior work on the individual level. Using the Lost at Sea social psychology task, we conduct a large-scale online experiment (N=748), randomly assigning groups to leader elections with either visible demographic attributes (e.g. name, gender) or pseudonymous aliases. We then simulate matched LLM groups conditioned on the human data, benchmarking Gemini 2.5, GPT 4.1, Claude Haiku 3.5, and Gemma 3. LLM behaviors diverge: some mirror human biases; others mask these biases and attempt to compensate for them. We empirically demonstrate that human-AI alignment in collective reasoning depends on context, cues, and model-specific inductive biases. Understanding how LLMs align with collective human behavior is critical to advancing socially-aligned AI, and demands dynamic benchmarks that capture the complexities of collective reasoning.