Evaluating Factuality in Generation with Dependency-level Entailment
This addresses the issue of factual errors in generated text for applications like summarization and paraphrasing, representing an incremental improvement over existing methods.
The paper tackled the problem of factuality in text generation by proposing a dependency-level entailment formulation to identify and localize factual inconsistencies, showing that their model outperforms sentence-level methods and question generation approaches in paraphrasing and summarization tasks.
Despite significant progress in text generation models, a serious limitation is their tendency to produce text that is factually inconsistent with information in the input. Recent work has studied whether textual entailment systems can be used to identify factual errors; however, these sentence-level entailment models are trained to solve a different problem than generation filtering and they do not localize which part of a generation is non-factual. In this paper, we propose a new formulation of entailment that decomposes it at the level of dependency arcs. Rather than focusing on aggregate decisions, we instead ask whether the semantic relationship manifested by individual dependency arcs in the generated output is supported by the input. Human judgments on this task are difficult to obtain; we therefore propose a method to automatically create data based on existing entailment or paraphrase corpora. Experiments show that our dependency arc entailment model trained on this data can identify factual inconsistencies in paraphrasing and summarization better than sentence-level methods or those based on question generation, while additionally localizing the erroneous parts of the generation.