SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation
This addresses the problem of inefficient manual survey creation for researchers in computer science, but it is incremental as it focuses on benchmarking rather than a new generation method.
The authors tackled the lack of standardized benchmarks for automated scientific survey generation by introducing SurGE, a benchmark with test instances and an evaluation framework, revealing that advanced LLM-based methods still struggle significantly with this task.
The rapid growth of academic literature makes the manual creation of scientific surveys increasingly infeasible. While large language models show promise for automating this process, progress in this area is hindered by the absence of standardized benchmarks and evaluation protocols. To bridge this critical gap, we introduce SurGE (Survey Generation Evaluation), a new benchmark for scientific survey generation in computer science. SurGE consists of (1) a collection of test instances, each including a topic description, an expert-written survey, and its full set of cited references, and (2) a large-scale academic corpus of over one million papers. In addition, we propose an automated evaluation framework that measures the quality of generated surveys across four dimensions: comprehensiveness, citation accuracy, structural organization, and content quality. Our evaluation of diverse LLM-based methods demonstrates a significant performance gap, revealing that even advanced agentic frameworks struggle with the complexities of survey generation and highlighting the need for future research in this area. We have open-sourced all the code, data, and models at: https://github.com/oneal2000/SurGE