Large Language Models are Perplexed by some Political Parties
This work identifies a systematic political bias in LLMs for NLP practitioners and policymakers, but the finding is incremental as it extends known translation fairness patterns to perplexity.
The study evaluates political fairness of LLMs using perplexity, finding that models are more perplexed by texts from far-right and nationalist parties than social-democratic ones across 10 LLMs and 37 languages, with instruction tuning having little effect.
Large Language Models (LLMs) are increasingly used, including in political applications, but their political fairness has been little studied. We assess it using perplexity, posing that a fair model should give equal probability to all political groups. However, we find, across ten LLMs and three datasets covering 37 languages, that LLMs are more perplexed by the texts of far right and nationalist parties than of social-democratic parties. We find this to be consistent with previous work on translation fairness, to the point that perplexity correlates with downstream translation metrics. Our method is applicable to both base LLMs as well as their instruction-tuned counterpart, and we find that both are highly correlated, suggesting that the political fairness of LLMs stems from their pretraining, and is hardly affected by instruction-tuning.