Continual Distillation Learning: Knowledge Distillation in Prompt-based Continual Learning
This work addresses the challenge of inference efficiency in prompt-based continual learning for vision tasks, though it appears incremental as it adapts knowledge distillation to a specific context.
The paper tackles the problem of improving prompt-based continual learning models by using knowledge distillation from a large to a small vision transformer, introducing a novel method called Knowledge Distillation based on Prompts (KDP) that outperforms existing distillation techniques in this setup.
We introduce the problem of continual distillation learning (CDL) in order to use knowledge distillation (KD) to improve prompt-based continual learning (CL) models. The CDL problem is valuable to study since the use of a larger vision transformer (ViT) leads to better performance in prompt-based continual learning. The distillation of knowledge from a large ViT to a small ViT improves the inference efficiency for prompt-based CL models. We empirically found that existing KD methods such as logit distillation and feature distillation cannot effectively improve the student model in the CDL setup. To address this issue, we introduce a novel method named Knowledge Distillation based on Prompts (KDP), in which globally accessible prompts specifically designed for knowledge distillation are inserted into the frozen ViT backbone of the student model. We demonstrate that our KDP method effectively enhances the distillation performance in comparison to existing KD methods in the CDL setup.