With a Little Push, NLI Models can Robustly and Efficiently Predict Faithfulness
This addresses the need for reliable automatic faithfulness metrics in applications like summarization or human-machine interaction, offering an incremental improvement over existing methods.
The paper tackled the problem of unfaithful output from conditional language models by showing that pure NLI models can outperform more complex faithfulness metrics when enhanced with task-adaptive data augmentation and robust inference procedures, achieving strong improvements on the TRUE benchmark with favorable computational cost.
Conditional language models still generate unfaithful output that is not supported by their input. These unfaithful generations jeopardize trust in real-world applications such as summarization or human-machine interaction, motivating a need for automatic faithfulness metrics. To implement such metrics, NLI models seem attractive, since they solve a strongly related task that comes with a wealth of prior research and data. But recent research suggests that NLI models require costly additional machinery to perform reliably across datasets, e.g., by running inference on a cartesian product of input and generated sentences, or supporting them with a question-generation/answering step. In this work we show that pure NLI models _can_ outperform more complex metrics when combining task-adaptive data augmentation with robust inference procedures. We propose: (1) Augmenting NLI training data to adapt NL inferences to the specificities of faithfulness prediction in dialogue; (2) Making use of both entailment and contradiction probabilities in NLI, and (3) Using Monte-Carlo dropout during inference. Applied to the TRUE benchmark, which combines faithfulness datasets across diverse domains and tasks, our approach strongly improves a vanilla NLI model and significantly outperforms previous work, while showing favourable computational cost.