A Large-Scale Multi-Dimensional Empirical Study of LLMs for Conversation Summarization
For researchers and practitioners deploying LLMs for conversation summarization, this provides a comprehensive evaluation framework and practical guidance on model selection, though the insights are largely incremental.
The paper introduces OmniCSEval, a benchmark with 1,800 conversations across six scenarios and lengths up to 32k tokens, evaluating 28 LLMs on conversation summarization. Key findings include that reasoning models improve faithfulness but not completeness, and that small models can match large ones in specific scenarios.
Despite the significant advancement of LLMs in conversation summarization, their evaluation remains limited by insufficient scenarios, input lengths, and sample sizes. Furthermore, existing benchmarks often omit frontier reasoning systems and efficient small models, or lack fine-grained, multi-dimensional assessments. To bridge these gaps, we propose OmniCSEval, a unified benchmark comprising 1,800 diverse conversations across six real-world scenarios, featuring context lengths ranging from 128 to 32k tokens. For fine-grained evaluation, we employ a bidirectional fact-checking framework that integrates key fact matching to assess completeness and conciseness, alongside summary fact verification to evaluate faithfulness. To ensure reliable assessment, we establish a human-LLM collaborative pipeline for key fact extraction and a multi-LLM consensus verifier for summary fact decomposition. Leveraging this framework, we evaluate 28 LLMs across four distinct categories grouped by reasoning capability and model scale. Our extensive empirical study reveals critical insights regarding the cross-scenario challenges current LLMs continue to face, the impacts of reasoning and scale, and the efficiency and adaptability of reasoning models. We also provide guidance for system selection in real-world deployments.