Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks
This addresses the problem of computational efficiency and data privacy for real-world applications by enabling small models to handle knowledge-intensive tasks, though it is incremental as it builds on existing distillation and retrieval techniques.
The paper tackles the challenge of deploying large language models (LLMs) in knowledge-intensive reasoning tasks by proposing Knowledge-Augmented Reasoning Distillation (KARD), a method that fine-tunes small language models to generate rationales from LLMs with augmented external knowledge, resulting in 250M parameter models outperforming 3B parameter models on benchmarks like MedQA-USMLE and StrategyQA.
Large Language Models (LLMs) have shown promising performance in knowledge-intensive reasoning tasks that require a compound understanding of knowledge. However, deployment of the LLMs in real-world applications can be challenging due to their high computational requirements and concerns on data privacy. Previous studies have focused on building task-specific small Language Models (LMs) by fine-tuning them with labeled data or distilling LLMs. However, these approaches are ill-suited for knowledge-intensive reasoning tasks due to the limited capacity of small LMs in memorizing the knowledge required. Motivated by our theoretical analysis on memorization, we propose Knowledge-Augmented Reasoning Distillation (KARD), a novel method that fine-tunes small LMs to generate rationales obtained from LLMs with augmented knowledge retrieved from an external knowledge base. Moreover, we further propose a neural reranker to obtain documents relevant to rationale generation. We empirically show that KARD significantly improves the performance of small T5 and GPT models on the challenging knowledge-intensive reasoning datasets, namely MedQA-USMLE, StrategyQA, and OpenbookQA. Notably, our method makes the 250M T5 models achieve superior performance against the fine-tuned 3B models, having 12 times larger parameters, on both MedQA-USMLE and StrategyQA benchmarks.