Measuring Political Stance and Consistency in Large Language Models
This work addresses the problem of political bias and inconsistency in LLMs for users seeking information, though it is incremental as it builds on existing bias assessment methods.
The study assessed political stances of nine large language models on 24 sensitive issues using five prompting techniques, finding that models often adopt opposing stances, with Grok-3-mini being the most persistent and Mistral-7B the least, and no technique altered stances on issues like the Qatar blockade or oppression of Palestinians.
With the incredible advancements in Large Language Models (LLMs), many people have started using them to satisfy their information needs. However, utilizing LLMs might be problematic for political issues where disagreement is common and model outputs may reflect training-data biases or deliberate alignment choices. To better characterize such behavior, we assess the stances of nine LLMs on 24 politically sensitive issues using five prompting techniques. We find that models often adopt opposing stances on several issues; some positions are malleable under prompting, while others remain stable. Among the models examined, Grok-3-mini is the most persistent, whereas Mistral-7B is the least. For issues involving countries with different languages, models tend to support the side whose language is used in the prompt. Notably, no prompting technique alters model stances on the Qatar blockade or the oppression of Palestinians. We hope these findings raise user awareness when seeking political guidance from LLMs and encourage developers to address these concerns.