LUNA: A Framework for Language Understanding and Naturalness Assessment
This provides a convenient tool for researchers and practitioners in NLG to streamline evaluation, though it is incremental as it unifies existing metrics without introducing new ones.
The paper tackles the challenge of evaluating Natural Language Generation models by introducing LUNA, a unified framework that integrates 20 existing metrics categorized by reference-dependence and text representation, resulting in a user-friendly tool that simplifies evaluation with minimal code for extensions.
The evaluation of Natural Language Generation (NLG) models has gained increased attention, urging the development of metrics that evaluate various aspects of generated text. LUNA addresses this challenge by introducing a unified interface for 20 NLG evaluation metrics. These metrics are categorized based on their reference-dependence and the type of text representation they employ, from string-based n-gram overlap to the utilization of static embeddings and pre-trained language models. The straightforward design of LUNA allows for easy extension with novel metrics, requiring just a few lines of code. LUNA offers a user-friendly tool for evaluating generated texts.