GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
This provides a standardized and extensible evaluation framework for researchers and developers in multilingual natural language generation, though it is incremental as it builds on previous benchmarking efforts.
The authors tackled the problem of suboptimal evaluation practices in natural language generation by introducing GEMv2, a modular benchmark that supports 40 datasets in 51 languages, enabling easier online model evaluation and dataset integration.
Evaluation in machine learning is usually informed by past choices, for example which datasets or metrics to use. This standardization enables the comparison on equal footing using leaderboards, but the evaluation choices become sub-optimal as better alternatives arise. This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims. To make following best model evaluation practices easier, we introduce GEMv2. The new version of the Generation, Evaluation, and Metrics Benchmark introduces a modular infrastructure for dataset, model, and metric developers to benefit from each others work. GEMv2 supports 40 documented datasets in 51 languages. Models for all datasets can be evaluated online and our interactive data card creation and rendering tools make it easier to add new datasets to the living benchmark.