GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Model
This addresses the problem of memory and flexibility limitations for developers in industrial applications, though it is incremental as it builds on existing distillation methods.
The paper tackles the challenge of deploying knowledge distillation for large-scale pre-trained language models (over 10B parameters) in industrial settings, proposing GKD, a general framework that supports distillation on models up to 100B scale and 25 methods using 8 NVIDIA A100 GPUs.
Currently, the reduction in the parameter scale of large-scale pre-trained language models (PLMs) through knowledge distillation has greatly facilitated their widespread deployment on various devices. However, the deployment of knowledge distillation systems faces great challenges in real-world industrial-strength applications, which require the use of complex distillation methods on even larger-scale PLMs (over 10B), limited by memory on GPUs and the switching of methods. To overcome these challenges, we propose GKD, a general knowledge distillation framework that supports distillation on larger-scale PLMs using various distillation methods. With GKD, developers can build larger distillation models on memory-limited GPUs and easily switch and combine different distillation methods within a single framework. Experimental results show that GKD can support the distillation of at least 100B-scale PLMs and 25 mainstream methods on 8 NVIDIA A100 (40GB) GPUs.