Measuring Political Bias in Large Language Models: What Is Said and How It Is Said
This addresses the need for transparency in LLMs to mitigate polarization and harms in applications, though it is incremental as it extends bias measurement from gender and racial to political domains.
The paper tackled the problem of political bias in large language models by proposing a framework that measures bias through both content and style analysis on issues like reproductive rights and climate change, and demonstrated its scalability and explainability by evaluating eleven open-source LLMs.
We propose to measure political bias in LLMs by analyzing both the content and style of their generated content regarding political issues. Existing benchmarks and measures focus on gender and racial biases. However, political bias exists in LLMs and can lead to polarization and other harms in downstream applications. In order to provide transparency to users, we advocate that there should be fine-grained and explainable measures of political biases generated by LLMs. Our proposed measure looks at different political issues such as reproductive rights and climate change, at both the content (the substance of the generation) and the style (the lexical polarity) of such bias. We measured the political bias in eleven open-sourced LLMs and showed that our proposed framework is easily scalable to other topics and is explainable.