On Learning to Summarize with Large Language Models as References
This addresses the challenge of enhancing summarization models for researchers and practitioners, but it is incremental as it builds on existing LLM capabilities.
The study tackled the problem of improving smaller text summarization models by using large language models (LLMs) as references, finding that models trained with LLM supervision achieved significant performance gains in both LLM and human evaluations, with contrastive learning outperforming standard fine-tuning. For example, it showed improvements in evaluation metrics, though specific numbers were not provided.
Recent studies have found that summaries generated by large language models (LLMs) are favored by human annotators over the original reference summaries in commonly used summarization datasets. Therefore, we study an LLM-as-reference learning setting for smaller text summarization models to investigate whether their performance can be substantially improved. To this end, we use LLMs as both oracle summary generators for standard supervised fine-tuning and oracle summary evaluators for efficient contrastive learning that leverages the LLMs' supervision signals. We conduct comprehensive experiments with source news articles and find that (1) summarization models trained under the LLM-as-reference setting achieve significant performance improvement in both LLM and human evaluations; (2) contrastive learning outperforms standard supervised fine-tuning under both low and high resource settings. Our experimental results also enable a meta-analysis of LLMs' summary evaluation capacities under a challenging setting, showing that LLMs are not well-aligned with human evaluators. Particularly, our expert human evaluation reveals remaining nuanced performance gaps between LLMs and our fine-tuned models, which LLMs fail to capture. Thus, we call for further studies into both the potential and challenges of using LLMs in summarization model development.