Understanding Finetuning for Factual Knowledge Extraction
This work addresses the problem of optimizing fine-tuning for factual accuracy in knowledge-intensive tasks, revealing that data selection based on pretrained knowledge storage is crucial, though it is incremental in improving fine-tuning strategies.
The study investigates how fine-tuning on lesser-known facts, which are poorly stored during pretraining, leads to worse factuality in question answering tasks compared to fine-tuning on well-known facts, with deteriorations of 5-10% on benchmarks like PopQA, Entity Questions, and MMLU.
In this work, we study the impact of QA fine-tuning data on downstream factuality. We show that fine-tuning on lesser-known facts that are poorly stored during pretraining yields significantly worse factuality than fine-tuning on well-known facts, even when all facts are seen during pretraining. We prove this phenomenon theoretically, showing that training on lesser-known facts can lead the model to ignore subject entity names and instead output a generic plausible response even when the relevant factual knowledge is encoded in the model. On three question answering benchmarks (PopQA, Entity Questions, and MMLU) and two language models (Llama-2-7B and Mistral-7B), we find that (i) finetuning on a completely factual but lesser-known subset of the data deteriorates downstream factuality (5-10%) and (ii) finetuning on a subset of better-known examples matches or outperforms finetuning on the entire dataset. Ultimately, our results shed light on the interaction between pretrained knowledge and finetuning data and demonstrate the importance of taking into account how facts are stored in the pretrained model when fine-tuning for knowledge-intensive tasks.