Optimizing the role of human evaluation in LLM-based spoken document summarization systems
This work addresses the problem of unreliable automatic evaluations for LLM-based summarization systems, providing guidelines for researchers and practitioners to improve human evaluation practices, though it is incremental as it builds on existing evaluation methods.
The paper tackles the challenge of evaluating LLM-generated spoken document summaries by proposing a human-in-the-loop evaluation paradigm tailored for generative AI, drawing on social science methodologies to ensure robustness and replicability, with case studies from a major U.S. tech company.
The emergence of powerful LLMs has led to a paradigm shift in abstractive summarization of spoken documents. The properties that make LLMs so valuable for this task -- creativity, ability to produce fluent speech, and ability to abstract information from large corpora -- also present new challenges to evaluating their content. Quick, cost-effective automatic evaluations such as ROUGE and BERTScore offer promise, but do not yet show competitive performance when compared to human evaluations. We draw on methodologies from the social sciences to propose an evaluation paradigm for spoken document summarization explicitly tailored for generative AI content. We provide detailed evaluation criteria and best practices guidelines to ensure robustness in the experimental design, replicability, and trustworthiness of human evaluation studies. We additionally include two case studies that show how these human-in-the-loop evaluation methods have been implemented at a major U.S. technology company.