The Curious Case of Factuality Finetuning: Models' Internal Beliefs Can Improve Factuality
This addresses the challenge of mitigating factual errors in language models for practitioners, offering a novel approach that leverages models' internal beliefs, though it is incremental in the broader context of hallucination reduction.
The study tackled the problem of reducing hallucinations in language models by investigating finetuning data strategies, finding that using model-generated data filtered by the models' own internal judgments improves factuality more effectively than using factual gold data or other configurations.
Language models are prone to hallucination - generating text that is factually incorrect. Finetuning models on high-quality factual information can potentially reduce hallucination, but concerns remain; obtaining factual gold data can be expensive and training on correct but unfamiliar data may potentially lead to even more downstream hallucination. What data should practitioners finetune on to mitigate hallucinations in language models? In this work, we study the relationship between the factuality of finetuning data and the prevalence of hallucinations in long-form generation tasks. Counterintuitively, we find that finetuning on factual gold data is not as helpful as finetuning on model-generated data that models believe to be factual. Next, we evaluate filtering strategies applied on both factual gold data and model-generated data, and find that finetuning on model-generated data that is filtered by models' own internal judgments often leads to better overall factuality compared to other configurations: training on gold data filtered by models' judgments, training on gold data alone, or training on model-generated data that is supported by gold data. These factuality improvements transfer across three domains we study, suggesting that a models' own beliefs can provide a powerful signal for factuality.