Survey Response Generation: Generating Closed-Ended Survey Responses In-Silico with Large Language Models
This work addresses the challenge of standardizing in-silico survey simulations for researchers, though it is incremental as it builds on existing methods without introducing a new paradigm.
The paper tackled the problem of generating closed-ended survey responses with large language models, finding that restricted generation methods performed best overall and reasoning output did not consistently improve alignment, based on 32 million simulated responses across various methods and models.
Many in-silico simulations of human survey responses with large language models (LLMs) focus on generating closed-ended survey responses, whereas LLMs are typically trained to generate open-ended text instead. Previous research has used a diverse range of methods for generating closed-ended survey responses with LLMs, and a standard practice remains to be identified. In this paper, we systematically investigate the impact that various Survey Response Generation Methods have on predicted survey responses. We present the results of 32 mio. simulated survey responses across 8 Survey Response Generation Methods, 4 political attitude surveys, and 10 open-weight language models. We find significant differences between the Survey Response Generation Methods in both individual-level and subpopulation-level alignment. Our results show that Restricted Generation Methods perform best overall, and that reasoning output does not consistently improve alignment. Our work underlines the significant impact that Survey Response Generation Methods have on simulated survey responses, and we develop practical recommendations on the application of Survey Response Generation Methods.