Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos
This work provides a controlled methodology for auditing social bias in VLMs, addressing a critical need for fairness in high-stakes deployments.
The authors propose a face-only counterfactual evaluation paradigm to measure social bias in VLMs, isolating demographic effects from visual confounds. Using their FOCUS dataset and REFLECT benchmark, they find that demographic disparities persist across five state-of-the-art VLMs and vary by task formulation.
Vision-Language Models (VLMs) are increasingly deployed in socially consequential settings, raising concerns about social bias driven by demographic cues. A central challenge in measuring such social bias is attribution under visual confounding: real-world images entangle race and gender with correlated factors such as background and clothing, obscuring attribution. We propose a \textbf{face-only counterfactual evaluation paradigm} that isolates demographic effects while preserving real-image realism. Starting from real photographs, we generate counterfactual variants by editing only facial attributes related to race and gender, keeping all other visual factors fixed. Based on this paradigm, we construct \textbf{FOCUS}, a dataset of 480 scene-matched counterfactual images across six occupations and ten demographic groups, and propose \textbf{REFLECT}, a benchmark comprising three decision-oriented tasks: two-alternative forced choice, multiple-choice socioeconomic inference, and numeric salary recommendation. Experiments on five state-of-the-art VLMs reveal that demographic disparities persist under strict visual control and vary substantially across task formulations. These findings underscore the necessity of controlled, counterfactual audits and highlight task design as a critical factor in evaluating social bias in multimodal models.