Breaking Language Barriers: Equitable Performance in Multilingual Language Models
This addresses fairness for speakers of low-resource languages by reducing performance disparities in multilingual LLMs, though it is incremental as it builds on existing fine-tuning and code-switching methods.
The paper tackles the problem of LLMs performing worse in Common Sense Reasoning tasks for low-resource languages compared to high-resource languages by fine-tuning on synthetic code-switched text, resulting in substantial improvements in low-resource language performance while maintaining high-resource language performance.
Cutting-edge LLMs have emerged as powerful tools for multilingual communication and understanding. However, LLMs perform worse in Common Sense Reasoning (CSR) tasks when prompted in low-resource languages (LRLs) like Hindi or Swahili compared to high-resource languages (HRLs) like English. Equalizing this inconsistent access to quality LLM outputs is crucial to ensure fairness for speakers of LRLs and across diverse linguistic communities. In this paper, we propose an approach to bridge this gap in LLM performance. Our approach involves fine-tuning an LLM on synthetic code-switched text generated using controlled language-mixing methods. We empirically demonstrate that fine-tuning LLMs on synthetic code-switched datasets leads to substantial improvements in LRL model performance while preserving or enhancing performance in HRLs. Additionally, we present a new dataset of synthetic code-switched text derived from the CommonSenseQA dataset, featuring three distinct language ratio configurations.