Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Dataset
This addresses political fairness issues for users of multilingual LLMs in political contexts, but is incremental as it reframes bias assessment rather than introducing a new method.
The study tackled the problem of political bias in multilingual LLMs by assessing fairness in translation quality using a new 21-way multiparallel EuroParl dataset, finding systematic differences where majority parties (left, center, right) were better translated than outsider parties.
The political biases of Large Language Models (LLMs) are usually assessed by simulating their answers to English surveys. In this work, we propose an alternative framing of political biases, relying on principles of fairness in multilingual translation. We systematically compare the translation quality of speeches in the European Parliament (EP), observing systematic differences with majority parties from left, center, and right being better translated than outsider parties. This study is made possible by a new, 21-way multiparallel version of EuroParl, the parliamentary proceedings of the EP, which includes the political affiliations of each speaker. The dataset consists of 1.5M sentences for a total of 40M words and 249M characters. It covers three years, 1000+ speakers, 7 countries, 12 EU parties, 25 EU committees, and hundreds of national parties.