GlobalBench: A Benchmark for Global Progress in Natural Language Processing
This addresses the issue of uneven resource allocation and incentives in NLP for less resourced languages, aiming to promote equitable language technology globally, though it is incremental as it builds on prior multilingual benchmarks.
The authors tackled the problem of disparities in NLP system performance across languages by introducing GlobalBench, a dynamic benchmark that tracks progress on all NLP datasets in all languages, covering 966 datasets in 190 languages and 1,128 system submissions across 62 languages.
Despite the major advances in NLP, significant disparities in NLP system performance across languages still exist. Arguably, these are due to uneven resource allocation and sub-optimal incentives to work on less resourced languages. To track and further incentivize the global development of equitable language technology, we introduce GlobalBench. Prior multilingual benchmarks are static and have focused on a limited number of tasks and languages. In contrast, GlobalBench is an ever-expanding collection that aims to dynamically track progress on all NLP datasets in all languages. Rather than solely measuring accuracy, GlobalBench also tracks the estimated per-speaker utility and equity of technology across all languages, providing a multi-faceted view of how language technology is serving people of the world. Furthermore, GlobalBench is designed to identify the most under-served languages, and rewards research efforts directed towards those languages. At present, the most under-served languages are the ones with a relatively high population, but nonetheless overlooked by composite multilingual benchmarks (like Punjabi, Portuguese, and Wu Chinese). Currently, GlobalBench covers 966 datasets in 190 languages, and has 1,128 system submissions spanning 62 languages.