Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite Learning
This addresses hallucination issues in LLMs for applications requiring factual accuracy, though it appears incremental as it builds on existing fine-tuning methods.
The paper tackles the problem of LLM hallucinations caused by knowledge inconsistency between pre-training and fine-tuning by proposing Prereq-Tune, a fine-tuning strategy that disentangles skill and knowledge learning, and shows it outperforms existing baselines in improving factuality across QA and long-form generation tasks.
Recent studies have identified one aggravating factor of LLM hallucinations as the knowledge inconsistency between pre-training and fine-tuning, where unfamiliar fine-tuning data mislead the LLM to fabricate plausible but wrong outputs. In this paper, we propose a novel fine-tuning strategy called Prereq-Tune to address this knowledge inconsistency and reduce hallucinations. Fundamentally, Prereq-Tune disentangles the learning of skills and knowledge, so the model learns only the task skills without being impacted by the knowledge inconsistency. To achieve this, Prereq-Tune introduces an additional prerequisite learning stage to learn the necessary knowledge for SFT, allowing subsequent SFT to focus only on task skills. Prereq-Tune can also be combined with fictitious synthetic data to enhance the grounding of LLM outputs to their internal knowledge. Experiments show that Prereq-Tune outperforms existing baselines in improving LLM's factuality across short QA and long-form generation tasks. It also opens new possibilities for knowledge-controlled generation in LLMs. Our code is available at https://github.com/UCSB-NLP-Chang/Prereq_tune.git.