GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation
This work addresses the problem of efficient deployment of language models for NLP practitioners, offering a task-agnostic framework that improves generalizability, though it is incremental as it builds on existing knowledge distillation methods.
The paper tackles the problem of knowledge distillation from LLMs by addressing the limitation of generating data mainly from the center of the distribution, which hinders learning the true underlying data distribution. The result is that GOLD outperforms prior arts and the LLM with average improvements of 5% and 14% across 10 NLP tasks.
Knowledge distillation from LLMs is essential for the efficient deployment of language models. Prior works have proposed data generation using LLMs for preparing distilled models. We argue that generating data with LLMs is prone to sampling mainly from the center of original content distribution. This limitation hinders the distilled model from learning the true underlying data distribution and to forget the tails of the distributions (samples with lower probability). To this end, we propose GOLD, a task-agnostic data generation and knowledge distillation framework, which employs an iterative out-of-distribution-guided feedback mechanism for the LLM. As a result, the generated data improves the generalizability of distilled models. An energy-based OOD evaluation approach is also introduced to deal with noisy generated data. Our extensive experiments on 10 different classification and sequence-to-sequence tasks in NLP show that GOLD respectively outperforms prior arts and the LLM with an average improvement of 5% and 14%. We will also show that the proposed method is applicable to less explored and novel tasks. The code is available.