Dolphin: A Challenging and Diverse Benchmark for Arabic NLG
This addresses the problem of limited evaluation resources for Arabic NLG, enabling researchers to better assess and improve models, though it is incremental as it builds on existing datasets.
The authors tackled the lack of a comprehensive evaluation framework for Arabic natural language generation by introducing Dolphin, a benchmark covering 13 NLG tasks across 40 datasets and 50 test splits, which sets new baselines for model performance.
We present Dolphin, a novel benchmark that addresses the need for a natural language generation (NLG) evaluation framework dedicated to the wide collection of Arabic languages and varieties. The proposed benchmark encompasses a broad range of 13 different NLG tasks, including dialogue generation, question answering, machine translation, summarization, among others. Dolphin comprises a substantial corpus of 40 diverse and representative public datasets across 50 test splits, carefully curated to reflect real-world scenarios and the linguistic richness of Arabic. It sets a new standard for evaluating the performance and generalization capabilities of Arabic and multilingual models, promising to enable researchers to push the boundaries of current methodologies. We provide an extensive analysis of Dolphin, highlighting its diversity and identifying gaps in current Arabic NLG research. We also offer a public leaderboard that is both interactive and modular and evaluate several models on our benchmark, allowing us to set strong baselines against which researchers can compare.