CaPE: Contrastive Parameter Ensembling for Reducing Hallucination in Abstractive Summarization
This addresses hallucination issues for users of abstractive summarization models, representing an incremental improvement by refining training data usage.
The paper tackles hallucination in neural abstractive summarization by proposing Contrastive Parameter Ensembling (CaPE), which uses clean and noisy training subsets to adjust model parameters, resulting in improvements of up to 16.69% and 15.78% in factual accuracy on XSUM and CNN/DM datasets without degrading other metrics.
Hallucination is a known issue for neural abstractive summarization models. Recent work suggests that the degree of hallucination may depend on errors in the training data. In this work, we propose a new method called Contrastive Parameter Ensembling (CaPE) to use training data more effectively, utilizing variations in noise in training samples to reduce hallucination. We first select clean and noisy subsets from the training data using different automatic factual metrics. Then, we fine-tune a base summarization model, which is trained on all training samples, on the clean (noisy) subset to obtain an \textit{expert} (\textit{anti-expert}) model. Finally, we adjust the parameters of base model by the difference between parameters of the \textit{expert} and \textit{anti-expert} models, steering the base model towards the \textit{expert} model and away from the \textit{anti-expert} model. Experimental results show that CaPE improves performance across different automatic factual metrics and human evaluation, with the maximum improvement of 16.69\% and 15.78\% on summary-level dependency-arc entailment accuracy for the XSUM and CNN/DM datasets. The improvement in factual performance does not degrade the performance on other metrics of informativeness such as ROUGE.