Not All Languages are Equal: Insights into Multilingual Retrieval-Augmented Generation
This work addresses the challenge of ensuring equitable performance across languages in RALMs, which is crucial for global AI applications, but it is incremental as it primarily provides insights and recommendations based on benchmarking.
The paper tackles the problem of linguistic inequalities in multilingual retrieval-augmented language models (RALMs) by evaluating six models on a new benchmark, Futurepedia, across eight languages, revealing that high-resource and Indo-European languages perform better in knowledge extraction and transfer, while English dominates in knowledge selection.
RALMs (Retrieval-Augmented Language Models) broaden their knowledge scope by incorporating external textual resources. However, the multilingual nature of global knowledge necessitates RALMs to handle diverse languages, a topic that has received limited research focus. In this work, we propose \textit{Futurepedia}, a carefully crafted benchmark containing parallel texts across eight representative languages. We evaluate six multilingual RALMs using our benchmark to explore the challenges of multilingual RALMs. Experimental results reveal linguistic inequalities: 1) high-resource languages stand out in Monolingual Knowledge Extraction; 2) Indo-European languages lead RALMs to provide answers directly from documents, alleviating the challenge of expressing answers across languages; 3) English benefits from RALMs' selection bias and speaks louder in multilingual knowledge selection. Based on these findings, we offer advice for improving multilingual Retrieval Augmented Generation. For monolingual knowledge extraction, careful attention must be paid to cascading errors from translating low-resource languages into high-resource ones. In cross-lingual knowledge transfer, encouraging RALMs to provide answers within documents in different languages can improve transfer performance. For multilingual knowledge selection, incorporating more non-English documents and repositioning English documents can help mitigate RALMs' selection bias. Through comprehensive experiments, we underscore the complexities inherent in multilingual RALMs and offer valuable insights for future research.