Evaluating Factual Consistency of Summaries with Large Language Models
This addresses the challenge of factual consistency in summarization for NLP researchers, but it is incremental as it applies existing LLM prompting methods to a known task.
The paper tackled the problem of detecting factual errors in summaries by exploring the use of large language models (LLMs) as evaluators, and found that prompting LLMs outperformed previous best systems by up to 12.2 points in accuracy.
Detecting factual errors in summaries has been an important and challenging subject in summarization research. Inspired by the emergent ability of large language models (LLMs), we explore evaluating factual consistency of summaries by directly prompting LLMs. We present a comprehensive empirical study to assess the ability of LLMs as factual consistency evaluators, which consists of (1) analyzing different LLMs such as the GPT model series and Flan-T5; (2) investigating a variety of prompting methods including vanilla prompting, chain-of-thought prompting, and a sentence-by-sentence prompting method to tackle long summaries; and (3) evaluating on diverse summaries generated by multiple summarization systems, ranging from pre-transformer methods to SOTA pretrained models. Our experiments demonstrate that prompting LLMs is able to outperform the previous best factuality systems in all settings, by up to 12.2 absolute points in terms of the binary classification accuracy on inconsistency detection.