DeepSynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing
This addresses the need for better evaluation of synthesis capabilities in AI agents for deep research, though it is incremental as it builds on existing benchmarks for retrieval.
The paper tackles the problem of objectively evaluating how well AI agents consolidate information into long-form reports, by introducing DeepSynth-Eval, a benchmark that uses survey papers as gold standards and checklists for metrics. Results show that agentic workflows outperform single-turn generation, reducing hallucinations and improving structural adherence in 96 tasks.
The evolution of Large Language Models (LLMs) towards autonomous agents has catalyzed progress in Deep Research. While retrieval capabilities are well-benchmarked, the post-retrieval synthesis stage--where agents must digest massive amounts of context and consolidate fragmented evidence into coherent, long-form reports--remains under-evaluated due to the subjectivity of open-ended writing. To bridge this gap, we introduce DeepSynth-Eval, a benchmark designed to objectively evaluate information consolidation capabilities. We leverage high-quality survey papers as gold standards, reverse-engineering research requests and constructing "Oracle Contexts" from their bibliographies to isolate synthesis from retrieval noise. We propose a fine-grained evaluation protocol using General Checklists (for factual coverage) and Constraint Checklists (for structural organization), transforming subjective judgment into verifiable metrics. Experiments across 96 tasks reveal that synthesizing information from hundreds of references remains a significant challenge. Our results demonstrate that agentic plan-and-write workflows significantly outperform single-turn generation, effectively reducing hallucinations and improving adherence to complex structural constraints.