HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers
This addresses model compression for deploying pre-trained transformers in resource-constrained settings, but it is incremental as it builds on existing distillation methods.
The paper tackles the challenge of task-agnostic distillation for pre-trained language models, where large prediction discrepancies between teacher and student models reduce effectiveness, and proposes HomoDistil with iterative pruning to maintain small discrepancies, achieving significant improvements over baselines.
Knowledge distillation has been shown to be a powerful model compression approach to facilitate the deployment of pre-trained language models in practice. This paper focuses on task-agnostic distillation. It produces a compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. Despite the practical benefits, task-agnostic distillation is challenging. Since the teacher model has a significantly larger capacity and stronger representation power than the student model, it is very difficult for the student to produce predictions that match the teacher's over a massive amount of open-domain training data. Such a large prediction discrepancy often diminishes the benefits of knowledge distillation. To address this challenge, we propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning. Specifically, we initialize the student model from the teacher model, and iteratively prune the student's neurons until the target width is reached. Such an approach maintains a small discrepancy between the teacher's and student's predictions throughout the distillation process, which ensures the effectiveness of knowledge transfer. Extensive experiments demonstrate that HomoDistil achieves significant improvements on existing baselines.