Factual Dialogue Summarization via Learning from Large Language Models
This work addresses deployment challenges for dialogue summarization in resource-constrained settings, though it is incremental as it builds on existing distillation and contrastive learning techniques.
The paper tackles the problem of improving factual consistency in dialogue summarization for smaller pretrained models by using symbolic knowledge distillation from large language models, achieving better factual consistency while maintaining other quality metrics compared to strong baselines.
Factual consistency is an important quality in dialogue summarization. Large language model (LLM)-based automatic text summarization models generate more factually consistent summaries compared to those by smaller pretrained language models, but they face deployment challenges in real-world applications due to privacy or resource constraints. In this paper, we investigate the use of symbolic knowledge distillation to improve the factual consistency of smaller pretrained models for dialogue summarization. We employ zero-shot learning to extract symbolic knowledge from LLMs, generating both factually consistent (positive) and inconsistent (negative) summaries. We then apply two contrastive learning objectives on these summaries to enhance smaller summarization models. Experiments with BART, PEGASUS, and Flan-T5 indicate that our approach surpasses strong baselines that rely on complex data augmentation strategies. Our approach achieves better factual consistency while maintaining coherence, fluency, and relevance, as confirmed by various automatic evaluation metrics. We also provide access to the data and code to facilitate future research.