FAIR_XAI: Improving Multimodal Foundation Model Fairness via Explainability for Wellbeing Assessment
For researchers and practitioners deploying VLMs in clinical mental health settings, this work highlights the persistent trade-off between transparency and equitable outcomes, emphasizing the need for joint optimization of accuracy, fairness, and generalization.
The paper investigates fairness and diagnostic reliability of Vision-Language Models (VLMs) for wellbeing assessment, finding that performance varies widely across models and datasets (e.g., Phi3.5-Vision achieves 80.4% accuracy on E-DAIC while Qwen2-VL achieves 33.9%), and that explainability-based interventions can improve procedural consistency but often fail to ensure outcome fairness, sometimes worsening bias.
In recent years, the integration of multimodal machine learning in wellbeing assessment has offered transformative potential for monitoring mental health. However, with the rapid advancement of Vision-Language Models (VLMs), their deployment in clinical settings has raised concerns due to their lack of transparency and potential for bias. While previous research has explored the intersection of fairness and Explainable AI (XAI), its application to VLMs for wellbeing assessment and depression prediction remains under-explored. This work investigates VLM performance across laboratory (AFAR-BSFT) and naturalistic (E-DAIC) datasets, focusing on diagnostic reliability and demographic fairness. Performance varied substantially across environments and architectures; Phi3.5-Vision achieved 80.4% accuracy on E-DAIC, while Qwen2-VL struggled at 33.9%. Additionally, both models demonstrated a tendency to over-predict depression on AFAR-BSFT. Although bias existed across both architectures, Qwen2-VL showed higher gender disparities, while Phi-3.5-Vision exhibited more racial bias. Our XAI intervention framework yielded mixed results; fairness prompting achieved perfect equal opportunity for Qwen2-VL at a severe accuracy cost on E-DAIC. On AFAR-BSFT, explainability-based interventions improved procedural consistency but did not guarantee outcome fairness, sometimes amplifying racial bias. These results highlight a persistent gap between procedural transparency and equitable outcomes. We analyse these findings and consolidate concrete recommendations for addressing them, emphasising that future fairness interventions must jointly optimise predictive accuracy, demographic parity, and cross-domain generalisation.