The Democratic Paradox in Large Language Models' Underestimation of Press Freedom
This incremental research highlights biases in LLMs that could undermine public trust in democratic institutions like press freedom, affecting millions of users who rely on these models for information.
The study found that six popular LLMs systematically underestimate press freedom in 180 countries compared to expert assessments, with models rating 71% to 93% of countries as less free, and they disproportionately underestimate press freedom in countries where it is strongest, with home biases causing ratings 7% to 260% more positive for their home countries.
As Large Language Models (LLMs) increasingly mediate global information access for millions of users worldwide, their alignment and biases have the potential to shape public understanding and trust in fundamental democratic institutions, such as press freedom. In this study, we uncover three systematic distortions in the way six popular LLMs evaluate press freedom in 180 countries compared to expert assessments of the World Press Freedom Index (WPFI). The six LLMs exhibit a negative misalignment, consistently underestimating press freedom, with individual models rating between 71% to 93% of countries as less free. We also identify a paradoxical pattern we term differential misalignment: LLMs disproportionately underestimate press freedom in countries where it is strongest. Additionally, five of the six LLMs exhibit positive home bias, rating their home countries' press freedoms more favorably than would be expected given their negative misalignment with the human benchmark. In some cases, LLMs rate their home countries between 7% to 260% more positively than expected. If LLMs are set to become the next search engines and some of the most important cultural tools of our time, they must ensure accurate representations of the state of our human and civic rights globally.