Déjà Vu: Multilingual LLM Evaluation through the Lens of Machine Translation Evaluation
This work addresses the need for more comprehensive and rigorous evaluation methods in mLLM development, which is crucial for researchers and developers to guide progress in multilingual AI, though it is incremental by applying existing evaluation frameworks to a new domain.
The paper tackles the problem of inadequate evaluation practices for multilingual large language models (mLLMs) by drawing parallels with machine translation evaluation, demonstrating how its best practices can improve model quality assessment and providing a checklist for robust meta-evaluation.
Generation capabilities and language coverage of multilingual large language models (mLLMs) are advancing rapidly. However, evaluation practices for generative abilities of mLLMs are still lacking comprehensiveness, scientific rigor, and consistent adoption across research labs, which undermines their potential to meaningfully guide mLLM development. We draw parallels with machine translation (MT) evaluation, a field that faced similar challenges and has, over decades, developed transparent reporting standards and reliable evaluations for multilingual generative models. Through targeted experiments across key stages of the generative evaluation pipeline, we demonstrate how best practices from MT evaluation can deepen the understanding of quality differences between models. Additionally, we identify essential components for robust meta-evaluation of mLLMs, ensuring the evaluation methods themselves are rigorously assessed. We distill these insights into a checklist of actionable recommendations for mLLM research and development.