SurveyLens: A Research Discipline-Aware Benchmark for Automatic Survey Generation
This addresses the need for researchers, especially outside Computer Science, to evaluate ASG methods according to discipline-specific standards for generating high-quality surveys.
The paper tackles the problem that current Automatic Survey Generation (ASG) evaluation methods are biased toward Computer Science and lack discipline-specific standards, by introducing SurveyLens, a discipline-aware benchmark with a dataset of 1,000 human-written surveys across 10 disciplines and a dual-lens evaluation framework. The result is an analysis of 11 state-of-the-art ASG methods that reveals their distinct strengths and weaknesses across fields, providing guidance for tool selection.
The exponential growth of scientific literature has driven the evolution of Automatic Survey Generation (ASG) from simple pipelines to multi-agent frameworks and commercial Deep Research agents. However, current ASG evaluation methods rely on generic metrics and are heavily biased toward Computer Science (CS), failing to assess whether ASG methods adhere to the distinct standards of various academic disciplines. Consequently, researchers, especially those outside CS, lack clear guidance on using ASG systems to yield high-quality surveys compliant with specific discipline standards. To bridge this gap, we introduce SurveyLens, the first discipline-aware benchmark evaluating ASG methods across diverse research disciplines. We construct SurveyLens-1k, a curated dataset of 1,000 high-quality human-written surveys spanning 10 disciplines. Subsequently, we propose a dual-lens evaluation framework: (1) Discipline-Aware Rubric Evaluation, which utilizes LLMs with human preference-aligned weights to assess adherence to domain-specific writing standards; and (2) Canonical Alignment Evaluation to rigorously measure content coverage and synthesis quality against human-written survey papers. We conduct extensive experiments by evaluating 11 state-of-the-art ASG methods on SurveyLens, including Vanilla LLMs, ASG systems, and Deep Research agents. Our analysis reveals the distinct strengths and weaknesses of each paradigm across fields, providing essential guidance for selecting tools tailored to specific disciplinary requirements.