IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance
This addresses the risk of biased LLMs influencing user perspectives in writing assistance, providing a tool for robust measurement, though it is incremental as it builds on existing bias research.
The authors tackled the problem of measuring issue bias in LLM writing assistance by creating IssueBench, a dataset of 2.49 million realistic prompts based on 3.9k templates and 212 political issues, and found that issue biases are common and persistent across 10 state-of-the-art LLMs, with all models aligning more with US Democrat than Republican voter opinions.
Large language models (LLMs) are helping millions of users write texts about diverse issues, and in doing so expose users to different ideas and perspectives. This creates concerns about issue bias, where an LLM tends to present just one perspective on a given issue, which in turn may influence how users think about this issue. So far, it has not been possible to measure which issue biases LLMs manifest in real user interactions, making it difficult to address the risks from biased LLMs. Therefore, we create IssueBench: a set of 2.49m realistic English-language prompts to measure issue bias in LLM writing assistance, which we construct based on 3.9k templates (e.g. "write a blog about") and 212 political issues (e.g. "AI regulation") from real user interactions. Using IssueBench, we show that issue biases are common and persistent in 10 state-of-the-art LLMs. We also show that biases are very similar across models, and that all models align more with US Democrat than Republican voter opinion on a subset of issues. IssueBench can easily be adapted to include other issues, templates, or tasks. By enabling robust and realistic measurement, we hope that IssueBench can bring a new quality of evidence to ongoing discussions about LLM biases and how to address them.