$\mathcal{X}$-KD: General Experiential Knowledge Distillation for Large Language Models
This addresses the challenge of efficient knowledge transfer in large language models, offering a novel approach that improves performance and data efficiency, though it appears incremental by building on existing distillation methods.
The paper tackles the problem of knowledge distillation for large language models by proposing Experiential Knowledge Distillation (X-KD), a framework that enables student models to learn in the teacher's original learning environment, resulting in outperformance over baselines on tasks like abstractive summarization, machine translation, and arithmetic reasoning.
Knowledge Distillation (KD) for Large Language Models (LLMs) has become increasingly important as models grow in size and complexity. While existing distillation approaches focus on imitating teacher behavior, they often overlook the original learning environment that shaped the teacher's knowledge. Inspired by the experiential learning theory and inverse reinforcement learning, we propose Experiential Knowledge Distillation ($\mathcal{X}$-KD), a novel and general framework that enables student models to learn in the teacher's original learning environment. $\mathcal{X}$-KD adopts the Approximated Variational Reward Imitation Learning (AVRIL) framework to jointly model the teacher's original reward function and perform policy distillation, encouraging consistency between the student policy and the original reward function. Our derivation demonstrates that $\mathcal{X}$-KD follows the supervised learning framework and applies to both sequence-level and divergence-based distillation methods, underlining the simplicity and flexibility of our approach. Empirical results show that $\mathcal{X}$-KD outperforms the generalized KD and MiniLLM baselines on abstractive summarization, machine translation, and arithmetic reasoning tasks. Additionally, $\mathcal{X}$-KD achieves better performance-diversity trade-off and data efficiency than baseline KD approaches.