Data-Free Distillation of Language Model by Text-to-Text Transfer
This work addresses the challenge of model compression in NLP for scenarios where training data is unavailable, offering a novel approach that enhances specificity and diversity in generated data, though it is incremental as it builds on existing data-free distillation methods by extending them to generative models.
The paper tackles the problem of compressing language models without access to original training data by proposing a data-free knowledge distillation framework that uses a generative language model as a controllable data generator, resulting in improved distillation performance across various downstream tasks such as sentiment analysis and information extraction, with experiments showing it outperforms state-of-the-art methods.
Data-Free Knowledge Distillation (DFKD) plays a vital role in compressing the model when original training data is unavailable. Previous works for DFKD in NLP mainly focus on distilling encoder-only structures like BERT on classification tasks, which overlook the notable progress of generative language modeling. In this work, we propose a novel DFKD framework, namely DFKD-T$^{3}$, where the pretrained generative language model can also serve as a controllable data generator for model compression. This novel framework DFKD-T$^{3}$ leads to an end-to-end learnable text-to-text framework to transform the general domain corpus to compression-friendly task data, targeting to improve both the \textit{specificity} and \textit{diversity}. Extensive experiments show that our method can boost the distillation performance in various downstream tasks such as sentiment analysis, linguistic acceptability, and information extraction. Furthermore, we show that the generated texts can be directly used for distilling other language models and outperform the SOTA methods, making our method more appealing in a general DFKD setting. Our code is available at https://gitee.com/mindspore/models/tree/master/research/nlp/DFKD\_T3.