Don't Make It Up: Preserving Ignorance Awareness in LLM Fine-Tuning
This addresses a critical issue for LLM developers by preventing undesirable behaviors like hallucinations during fine-tuning, though it is incremental as it builds on existing fine-tuning approaches.
The paper tackles the problem of catastrophic forgetting in LLM fine-tuning, specifically the loss of ignorance awareness (epistemic uncertainty), and introduces SEAT, a method that preserves this capability while acquiring new knowledge, showing significant improvements over baselines in experiments.
Existing work on mitigating catastrophic forgetting during large language models (LLMs) fine-tuning for new knowledge instances has primarily focused on preserving performance on previously seen data, while critically overlooking the collapse of essential capabilities instilled through alignment, most notably the model's ability to faithfully express epistemic uncertainty (a property we term 'Ignorance Awareness'). In this work, we formalize the notion of Ignorance Awareness and illustrate that conventional fine-tuning methods can result in substantial activation displacement. This displacement undermines the critical capability of ignorance awareness, leading to undesirable behaviors such as hallucinations. To address this challenge, we introduce SEAT, a simple and principled fine-tuning approach that not only enables the model to effectively acquire new knowledge instances but also preserves its aligned ignorance awareness. SEAT integrates two key components: (1) sparse tuning that constrains activation drift, and (2) a novel entity perturbation method designed to counter knowledge entanglement. Experimental results demonstrate that, across both real-world and synthetic datasets, SEAT significantly outperforms baselines in preserving ignorance awareness while retaining optimal fine-tuning performance, offering a more robust solution for LLM fine-tuning.