Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages
This addresses the lack of quantitative evaluation methods for LLMs in low-resource languages, which is crucial for improving multilingual AI systems, though it is incremental as it builds on existing representation-based analysis.
The paper tackles the problem of uneven LLM performance across languages due to imbalanced training data by proposing the Language Ranker, a metric that quantifies and ranks languages based on LLM internal representations, revealing higher similarity scores for high-resource languages and a strong correlation with pre-training corpus proportions.
The development of Large Language Models (LLMs) relies on extensive text corpora, which are often unevenly distributed across languages. This imbalance results in LLMs performing significantly better on high-resource languages like English, German, and French, while their capabilities in low-resource languages remain inadequate. Currently, there is a lack of quantitative methods to evaluate the performance of LLMs in these low-resource languages. To address this gap, we propose the Language Ranker, an intrinsic metric designed to benchmark and rank languages based on LLM performance using internal representations. By comparing the LLM's internal representation of various languages against a baseline derived from English, we can assess the model's multilingual capabilities in a robust and language-agnostic manner. Our analysis reveals that high-resource languages exhibit higher similarity scores with English, demonstrating superior performance, while low-resource languages show lower similarity scores, underscoring the effectiveness of our metric in assessing language-specific capabilities. Besides, the experiments show that there is a strong correlation between the LLM's performance in different languages and the proportion of those languages in its pre-training corpus. These insights underscore the efficacy of the Language Ranker as a tool for evaluating LLM performance across different languages, particularly those with limited resources.