A Dual-Layered Evaluation of Geopolitical and Cultural Bias in LLMs
This addresses bias issues in LLMs for users in multilingual and culturally diverse contexts, offering a structured evaluation framework, though it is incremental in building on existing bias research.
The paper tackled the problem of bias in large language models (LLMs) by evaluating them on factual and disputable questions across multiple languages, finding that query language influences responses in factual scenarios and interacts with training context in sensitive disputes.
As large language models (LLMs) are increasingly deployed across diverse linguistic and cultural contexts, understanding their behavior in both factual and disputable scenarios is essential, especially when their outputs may shape public opinion or reinforce dominant narratives. In this paper, we define two types of bias in LLMs: model bias (bias stemming from model training) and inference bias (bias induced by the language of the query), through a two-phase evaluation. Phase 1 evaluates LLMs on factual questions where a single verifiable answer exists, assessing whether models maintain consistency across different query languages. Phase 2 expands the scope by probing geopolitically sensitive disputes, where responses may reflect culturally embedded or ideologically aligned perspectives. We construct a manually curated dataset spanning both factual and disputable QA, across four languages and question types. The results show that Phase 1 exhibits query language induced alignment, while Phase 2 reflects an interplay between the model's training context and query language. This paper offers a structured framework for evaluating LLM behavior across neutral and sensitive topics, providing insights for future LLM deployment and culturally aware evaluation practices in multilingual contexts.