GLGE: A New General Language Generation Evaluation Benchmark
This benchmark addresses the lack of comprehensive multi-task evaluation for Natural Language Generation (NLG) models, providing a standardized platform for researchers to assess and compare model generalization.
This paper introduces GLGE, a new multi-task benchmark for evaluating the generalization capabilities of Natural Language Generation (NLG) models across eight language generation tasks. It further designs three subtasks per task based on difficulty, resulting in 24 subtasks, and provides a public leaderboard with strong baselines.
Multi-task benchmarks such as GLUE and SuperGLUE have driven great progress of pretraining and transfer learning in Natural Language Processing (NLP). These benchmarks mostly focus on a range of Natural Language Understanding (NLU) tasks, without considering the Natural Language Generation (NLG) models. In this paper, we present the General Language Generation Evaluation (GLGE), a new multi-task benchmark for evaluating the generalization capabilities of NLG models across eight language generation tasks. For each task, we continue to design three subtasks in terms of task difficulty (GLGE-Easy, GLGE-Medium, and GLGE-Hard). This introduces 24 subtasks to comprehensively compare model performance. To encourage research on pretraining and transfer learning on NLG models, we make GLGE publicly available and build a leaderboard with strong baselines including MASS, BART, and ProphetNet (The source code and dataset are publicly available at https://github.com/microsoft/glge).