An Analysis of Large Language Models for Simulating User Responses in Surveys
This addresses the issue of biased or inaccurate user simulation in surveys for researchers and practitioners using LLMs, but it is incremental as it builds on known limitations without a breakthrough solution.
The paper tackled the problem of using Large Language Models (LLMs) to simulate human responses in surveys, finding that while a proposed method (CLAIMSIM) increased response diversity, both it and standard prompting approaches struggled to accurately simulate users, with LLMs showing fixed viewpoints and difficulty reasoning over demographic nuances.
Using Large Language Models (LLMs) to simulate user opinions has received growing attention. Yet LLMs, especially trained with reinforcement learning from human feedback (RLHF), are known to exhibit biases toward dominant viewpoints, raising concerns about their ability to represent users from diverse demographic and cultural backgrounds. In this work, we examine the extent to which LLMs can simulate human responses to cross-domain survey questions through direct prompting and chain-of-thought prompting. We further propose a claim diversification method CLAIMSIM, which elicits viewpoints from LLM parametric knowledge as contextual input. Experiments on the survey question answering task indicate that, while CLAIMSIM produces more diverse responses, both approaches struggle to accurately simulate users. Further analysis reveals two key limitations: (1) LLMs tend to maintain fixed viewpoints across varying demographic features, and generate single-perspective claims; and (2) when presented with conflicting claims, LLMs struggle to reason over nuanced differences among demographic features, limiting their ability to adapt responses to specific user profiles.