Can Machines Think Like Humans? A Behavioral Evaluation of LLM Agents in Dictator Games
This addresses the problem of understanding and evaluating prosocial behaviors in LLM agents for researchers and policymakers, but it is incremental as it applies existing social science methods to a new context.
The study investigated how different personas and experimental framings affect LLM agents' altruistic behavior in dictator games, comparing them across models and with humans, finding that assigning human-like identities does not produce human-like behaviors and alignment varies unpredictably.
As Large Language Model (LLM)-based agents increasingly engage with human society, how well do we understand their prosocial behaviors? We (1) investigate how LLM agents' prosocial behaviors can be induced by different personas and benchmarked against human behaviors; and (2) introduce a social science approach to evaluate LLM agents' decision-making. We explored how different personas and experimental framings affect these AI agents' altruistic behavior in dictator games and compared their behaviors within the same LLM family, across various families, and with human behaviors. The findings reveal that merely assigning a human-like identity to LLMs does not produce human-like behaviors. These findings suggest that LLM agents' reasoning does not consistently exhibit textual markers of human decision-making in dictator games and that their alignment with human behavior varies substantially across model architectures and prompt formulations; even worse, such dependence does not follow a clear pattern. As society increasingly integrates machine intelligence, "Prosocial AI" emerges as a promising and urgent research direction in philanthropic studies.