Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework
This work addresses the problem of model selection for researchers and practitioners in natural language processing, though it is incremental as it builds on existing evaluation metrics.
The paper tackles the challenge of evaluating open-ended text generation models by proposing a multicriteria evaluation framework that balances coherence, diversity, and perplexity, resulting in robust methods for ranking decoding strategies.
Open-ended text generation has become a prominent task in natural language processing due to the rise of powerful (large) language models. However, evaluating the quality of these models and the employed decoding strategies remains challenging due to trade-offs among widely used metrics such as coherence, diversity, and perplexity. This paper addresses the specific problem of multicriteria evaluation for open-ended text generation, proposing novel methods for both relative and absolute rankings of decoding methods. Specifically, we employ benchmarking approaches based on partial orderings and present a new summary metric to balance existing automatic indicators, providing a more holistic evaluation of text generation quality. Our experiments demonstrate that the proposed approaches offer a robust way to compare decoding strategies and serve as valuable tools to guide model selection for open-ended text generation tasks. We suggest future directions for improving evaluation methodologies in text generation and make our code, datasets, and models publicly available.