Questioning the Survey Responses of Large Language Models
This challenges the reliability of survey-based methods for assessing language models, showing results may be artifacts of biases rather than meaningful representations, which is important for researchers in AI ethics and evaluation.
The study examined the use of surveys to infer demographics or values from large language models, finding that model responses are biased by ordering and labeling (e.g., favoring 'A') and become uniformly random when biases are adjusted, contradicting prior claims about alignment.
Surveys have recently gained popularity as a tool to study large language models. By comparing survey responses of models to those of human reference populations, researchers aim to infer the demographics, political opinions, or values best represented by current language models. In this work, we critically examine this methodology on the basis of the well-established American Community Survey by the U.S. Census Bureau. Evaluating 43 different language models using de-facto standard prompting methodologies, we establish two dominant patterns. First, models' responses are governed by ordering and labeling biases, for example, towards survey responses labeled with the letter "A". Second, when adjusting for these systematic biases through randomized answer ordering, models across the board trend towards uniformly random survey responses, irrespective of model size or pre-training data. As a result, in contrast to conjectures from prior work, survey-derived alignment measures often permit a simple explanation: models consistently appear to better represent subgroups whose aggregate statistics are closest to uniform for any survey under consideration.