BIPOLAR: Polarization-based granular framework for LLM bias evaluation
This work addresses bias evaluation in LLMs for researchers and practitioners, offering a topic-agnostic framework, but it is incremental as it builds on existing bias detection methods.
The study tackled the problem of evaluating polarization-related biases in large language models by proposing a reusable, granular framework, and found that models like Llama-3, Mistral, GPT-4, Claude 3.5, and Gemini 1.0 showed a general trend of more positive sentiment toward Ukraine in a Russia-Ukraine war case study, with fine-grained analysis revealing considerable variation between semantic categories.
Large language models (LLMs) are known to exhibit biases in downstream tasks, especially when dealing with sensitive topics such as political discourse, gender identity, ethnic relations, or national stereotypes. Although significant progress has been made in bias detection and mitigation techniques, certain challenges remain underexplored. This study proposes a reusable, granular, and topic-agnostic framework to evaluate polarisation-related biases in LLM (both open-source and closed-source). Our approach combines polarisation-sensitive sentiment metrics with a synthetically generated balanced dataset of conflict-related statements, using a predefined set of semantic categories. As a case study, we created a synthetic dataset that focusses on the Russia-Ukraine war, and we evaluated the bias in several LLMs: Llama-3, Mistral, GPT-4, Claude 3.5, and Gemini 1.0. Beyond aggregate bias scores, with a general trend for more positive sentiment toward Ukraine, the framework allowed fine-grained analysis with considerable variation between semantic categories, uncovering divergent behavioural patterns among models. Adaptation to prompt modifications showed further bias towards preconceived language and citizenship modification. Overall, the framework supports automated dataset generation and fine-grained bias assessment, is applicable to a variety of polarisation-driven scenarios and topics, and is orthogonal to many other bias-evaluation strategies.