LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigm
This work addresses the evaluation gap in long-text generation for NLP researchers and developers, though it is incremental as it builds on existing paradigms.
The authors tackled the problem of evaluating long-text generation in large language models, revealing that performance deteriorates with increasing text length, and introduced LongEval, a benchmark showing that small-scale models trained on long texts can achieve comparable performance to larger models.
Large Language Models (LLMs) have achieved remarkable success in various natural language processing tasks, yet their ability to generate long-form content remains poorly understood and evaluated. Our analysis reveals that current LLMs struggle with length requirements and information density in long-text generation, with performance deteriorating as text length increases. To quantitively locate such a performance degradation and provide further insights on model development, we present LongEval, a benchmark that evaluates long-text generation through both direct and plan-based generation paradigms, inspired by cognitive and linguistic writing models. The comprehensive experiments in this work reveal interesting findings such as that while model size correlates with generation ability, the small-scale model (e.g., LongWriter), well-trained on long texts, has comparable performance. All code and datasets are released in https://github.com/Wusiwei0410/LongEval.