Replicating Human Motivated Reasoning Studies with LLMs
This highlights limitations for researchers using LLMs to automate tasks like survey data collection and argument assessment, indicating incremental insights into model-human discrepancies.
The study investigated whether base large language models (LLMs) replicate human motivated reasoning by replicating four prior political studies, finding that LLM behavior does not align with human patterns and shows consistent inaccuracies in argument strength assessments.
Motivated reasoning -- the idea that individuals processing information may be motivated to reach a certain conclusion, whether it be accurate or predetermined -- has been well-explored as a human phenomenon. However, it is unclear whether base LLMs mimic these motivational changes. Replicating 4 prior political motivated reasoning studies, we find that base LLM behavior does not align with expected human behavior. Furthermore, base LLM behavior across models shares some similarities, such as smaller standard deviations and inaccurate argument strength assessments. We emphasize the importance of these findings for researchers using LLMs to automate tasks such as survey data collection and argument assessment.