Representation Bias in Political Sample Simulations with Large Language Models
This addresses representation bias in AI for computational social science, offering incremental insights for researchers and practitioners.
The study quantified biases in simulating political samples using GPT-3.5-Turbo, finding that performance was better for vote choice than public opinions, more accurate in English-speaking countries, bipartisan systems, and democratic regimes.
This study seeks to identify and quantify biases in simulating political samples with Large Language Models, specifically focusing on vote choice and public opinion. Using the GPT-3.5-Turbo model, we leverage data from the American National Election Studies, German Longitudinal Election Study, Zuobiao Dataset, and China Family Panel Studies to simulate voting behaviors and public opinions. This methodology enables us to examine three types of representation bias: disparities based on the the country's language, demographic groups, and political regime types. The findings reveal that simulation performance is generally better for vote choice than for public opinions, more accurate in English-speaking countries, more effective in bipartisan systems than in multi-partisan systems, and stronger in democratic settings than in authoritarian regimes. These results contribute to enhancing our understanding and developing strategies to mitigate biases in AI applications within the field of computational social science.