Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning
This addresses a bias in AI models for users in diverse global contexts, though it is incremental as it builds on existing datasets and models.
The paper tackles the problem of vision-and-language models lacking cultural and geographic diversity in commonsense reasoning by constructing the Geo-Diverse Visual Commonsense Reasoning (GD-VCR) dataset, finding that models like VisualBERT and ViLBERT perform significantly worse on non-Western regions (e.g., East Asia, South Asia, Africa) compared to Western regions, with performance gaps up to significant percentages in culture-related scenarios.
Commonsense is defined as the knowledge that is shared by everyone. However, certain types of commonsense knowledge are correlated with culture and geographic locations and they are only shared locally. For example, the scenarios of wedding ceremonies vary across regions due to different customs influenced by historical and religious factors. Such regional characteristics, however, are generally omitted in prior work. In this paper, we construct a Geo-Diverse Visual Commonsense Reasoning dataset (GD-VCR) to test vision-and-language models' ability to understand cultural and geo-location-specific commonsense. In particular, we study two state-of-the-art Vision-and-Language models, VisualBERT and ViLBERT trained on VCR, a standard multimodal commonsense benchmark with images primarily from Western regions. We then evaluate how well the trained models can generalize to answering the questions in GD-VCR. We find that the performance of both models for non-Western regions including East Asia, South Asia, and Africa is significantly lower than that for Western region. We analyze the reasons behind the performance disparity and find that the performance gap is larger on QA pairs that: 1) are concerned with culture-related scenarios, e.g., weddings, religious activities, and festivals; 2) require high-level geo-diverse commonsense reasoning rather than low-order perception and recognition. Dataset and code are released at https://github.com/WadeYin9712/GD-VCR.