Leveraging Hierarchical Organization for Medical Multi-document Summarization
This work addresses the challenge of generating clear and preferred summaries for medical professionals, though it is incremental as it builds on existing LLM methods with structural modifications.
This paper tackled the problem of medical multi-document summarization by investigating whether hierarchical structures in inputs improve model-generated summaries compared to flat methods, finding that hierarchical approaches increase human preference while preserving factuality, coverage, and coherence.
Medical multi-document summarization (MDS) is a complex task that requires effectively managing cross-document relationships. This paper investigates whether incorporating hierarchical structures in the inputs of MDS can improve a model's ability to organize and contextualize information across documents compared to traditional flat summarization methods. We investigate two ways of incorporating hierarchical organization across three large language models (LLMs), and conduct comprehensive evaluations of the resulting summaries using automated metrics, model-based metrics, and domain expert evaluation of preference, understandability, clarity, complexity, relevance, coverage, factuality, and coherence. Our results show that human experts prefer model-generated summaries over human-written summaries. Hierarchical approaches generally preserve factuality, coverage, and coherence of information, while also increasing human preference for summaries. Additionally, we examine whether simulated judgments from GPT-4 align with human judgments, finding higher agreement along more objective evaluation facets. Our findings demonstrate that hierarchical structures can improve the clarity of medical summaries generated by models while maintaining content coverage, providing a practical way to improve human preference for generated summaries.