Efficient Knowledge Injection in LLMs via Self-Distillation
This addresses the challenge of updating LLMs with new knowledge for practical applications, offering a more efficient alternative to existing methods like fine-tuning and RAG, though it appears incremental as it adapts a known technique (self-distillation) to a new task.
The paper tackles the problem of efficiently injecting new factual knowledge into large language models (LLMs) without relying on larger teacher models or structured data, proposing a self-distillation-based method called prompt distillation that outperforms standard supervised fine-tuning and can surpass retrieval-augmented generation (RAG) across multiple LLM sizes and families.
In many practical applications, large language models (LLMs) need to acquire new knowledge not present in their pre-training data. Efficiently leveraging this knowledge usually relies on supervised fine-tuning or retrieval-augmented generation (RAG). Although RAG has emerged as the industry standard for knowledge injection, fine-tuning has not yet achieved comparable success. This paper proposes utilizing prompt distillation, a self-distillation-based method previously explored primarily for style alignment and instruction tuning, to internalize new factual knowledge from free-form documents. Unlike prior methods, our approach requires neither larger teacher models nor structured knowledge formats. Across multiple LLM sizes and model families, we show that prompt distillation outperforms standard supervised fine-tuning and can even surpass RAG. We analyze the key factors contributing to prompt distillation's effectiveness and examine how it scales.