Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomy
This addresses the issue of unreliable summaries for users of generative models, though it is incremental as it builds on existing NLI and LLM approaches.
The paper tackled the problem of factual inconsistencies in summaries generated by models by incorporating a task-specific error type taxonomy into LLM inference, achieving state-of-the-art performance in zero-shot settings across ten datasets and enabling efficient distillation of smaller models.
Factual inconsistencies pose a significant hurdle for the faithful summarization by generative models. While a major direction to enhance inconsistency detection is to derive stronger Natural Language Inference (NLI) models, we propose an orthogonal aspect that underscores the importance of incorporating task-specific taxonomy into the inference. To this end, we consolidate key error types of inconsistent facts in summaries, and incorporate them to facilitate both the zero-shot and supervised paradigms of LLMs. Extensive experiments on ten datasets of five distinct domains suggest that, zero-shot LLM inference could benefit from the explicit solution space depicted by the error type taxonomy, and achieves state-of-the-art performance overall, surpassing specialized non-LLM baselines, as well as recent LLM baselines. We further distill models that fuse the taxonomy into parameters through our designed prompt completions and supervised training strategies, efficiently substituting state-of-the-art zero-shot inference with much larger LLMs.