Interpreting Social Bias in LVLMs via Information Flow Analysis and Multi-Round Dialogue Evaluation
This work addresses the issue of understanding the underlying mechanisms of social bias in LVLMs, which is crucial for improving fairness in AI systems, though it is incremental as it builds on existing bias detection methods.
The study tackled the problem of social bias in Large Vision Language Models (LVLMs) by proposing an explanatory framework combining information flow analysis and multi-round dialogue evaluation, revealing systematic disparities in information usage across demographic groups and biased semantic representations in textual modalities.
Large Vision Language Models (LVLMs) have achieved remarkable progress in multimodal tasks, yet they also exhibit notable social biases. These biases often manifest as unintended associations between neutral concepts and sensitive human attributes, leading to disparate model behaviors across demographic groups. While existing studies primarily focus on detecting and quantifying such biases, they offer limited insight into the underlying mechanisms within the models. To address this gap, we propose an explanatory framework that combines information flow analysis with multi-round dialogue evaluation, aiming to understand the origin of social bias from the perspective of imbalanced internal information utilization. Specifically, we first identify high-contribution image tokens involved in the model's reasoning process for neutral questions via information flow analysis. Then, we design a multi-turn dialogue mechanism to evaluate the extent to which these key tokens encode sensitive information. Extensive experiments reveal that LVLMs exhibit systematic disparities in information usage when processing images of different demographic groups, suggesting that social bias is deeply rooted in the model's internal reasoning dynamics. Furthermore, we complement our findings from a textual modality perspective, showing that the model's semantic representations already display biased proximity patterns, thereby offering a cross-modal explanation of bias formation.