Annotations on a Budget: Leveraging Geo-Data Similarity to Balance Model Performance and Annotation Cost
This addresses the issue of poor model performance for underrepresented countries due to imbalanced training data, offering a cost-effective solution for improving global AI equity.
The paper tackles the problem of geographical bias in foundation models by proposing methods to identify which countries' data to annotate to improve performance for underrepresented regions while reducing annotation costs, showing that supplementing training with visually similar countries enhances model effectiveness.
Current foundation models have shown impressive performance across various tasks. However, several studies have revealed that these models are not effective for everyone due to the imbalanced geographical and economic representation of the data used in the training process. Most of this data comes from Western countries, leading to poor results for underrepresented countries. To address this issue, more data needs to be collected from these countries, but the cost of annotation can be a significant bottleneck. In this paper, we propose methods to identify the data to be annotated to balance model performance and annotation costs. Our approach first involves finding the countries with images of topics (objects and actions) most visually distinct from those already in the training datasets used by current large vision-language foundation models. Next, we identify countries with higher visual similarity for these topics and show that using data from these countries to supplement the training data improves model performance and reduces annotation costs. The resulting lists of countries and corresponding topics are made available at https://github.com/MichiganNLP/visual_diversity_budget.