Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings
This addresses the challenge of efficient natural language understanding for scenarios like mobile deployment where data is scarce, representing an incremental improvement over existing distillation techniques.
The paper tackles the problem of deploying large language models in low-data settings by proposing generation-distillation, which uses large models to generate training examples and distill knowledge into smaller networks, achieving comparable performance to BERT with 300x fewer parameters and outperforming prior distillation methods with 3x fewer parameters.
Over the past year, the emergence of transfer learning with large-scale language models (LM) has led to dramatic performance improvements across a broad range of natural language understanding tasks. However, the size and memory footprint of these large LMs makes them difficult to deploy in many scenarios (e.g. on mobile phones). Recent research points to knowledge distillation as a potential solution, showing that when training data for a given task is abundant, it is possible to distill a large (teacher) LM into a small task-specific (student) network with minimal loss of performance. However, when such data is scarce, there remains a significant performance gap between large pretrained LMs and smaller task-specific models, even when training via distillation. In this paper, we bridge this gap with a novel training approach, called generation-distillation, that leverages large finetuned LMs in two ways: (1) to generate new (unlabeled) training examples, and (2) to distill their knowledge into a small network using these examples. Across three low-resource text classification datsets, we achieve comparable performance to BERT while using 300x fewer parameters, and we outperform prior approaches to distillation for text classification while using 3x fewer parameters.