CILDA: Contrastive Data Augmentation using Intermediate Layer Knowledge Distillation
This work addresses the need for more efficient model compression in NLP, though it appears incremental as it builds on existing techniques like contrastive learning and intermediate layer distillation.
The paper tackled the problem of improving knowledge distillation for compressing pre-trained language models by proposing CILDA, a data augmentation technique that uses intermediate layer representations and contrastive loss, which outperformed state-of-the-art methods on the GLUE benchmark and in out-of-domain evaluations.
Knowledge distillation (KD) is an efficient framework for compressing large-scale pre-trained language models. Recent years have seen a surge of research aiming to improve KD by leveraging Contrastive Learning, Intermediate Layer Distillation, Data Augmentation, and Adversarial Training. In this work, we propose a learning based data augmentation technique tailored for knowledge distillation, called CILDA. To the best of our knowledge, this is the first time that intermediate layer representations of the main task are used in improving the quality of augmented samples. More precisely, we introduce an augmentation technique for KD based on intermediate layer matching using contrastive loss to improve masked adversarial data augmentation. CILDA outperforms existing state-of-the-art KD approaches on the GLUE benchmark, as well as in an out-of-domain evaluation.