Fidelity-Enriched Contrastive Search: Reconciling the Faithfulness-Diversity Trade-Off in Text Generation
This addresses the faithfulness-diversity trade-off in text generation for applications requiring accurate and varied outputs, representing an incremental advancement in decoding methods.
The paper tackled the hallucination problem in natural language generation by proposing Fidelity-Enriched Contrastive Search (FECS), a decoding method that enhances faithfulness to the source while maintaining diversity, with results showing consistent improvements across tasks like abstractive summarization and dialogue generation.
In this paper, we address the hallucination problem commonly found in natural language generation tasks. Language models often generate fluent and convincing content but can lack consistency with the provided source, resulting in potential inaccuracies. We propose a new decoding method called Fidelity-Enriched Contrastive Search (FECS), which augments the contrastive search framework with context-aware regularization terms. FECS promotes tokens that are semantically similar to the provided source while penalizing repetitiveness in the generated text. We demonstrate its effectiveness across two tasks prone to hallucination: abstractive summarization and dialogue generation. Results show that FECS consistently enhances faithfulness across various language model sizes while maintaining output diversity comparable to well-performing decoding algorithms.