PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation
This work addresses efficiency and performance issues in adapting pre-trained language models for researchers and practitioners, though it is incremental as it builds on existing PoT and knowledge distillation techniques.
The paper tackles the sub-optimal performance of Prompt Transfer (PoT) in prompt-tuning by proposing a metric to predict transferability and a method called PANDA that uses knowledge distillation to reduce knowledge forgetting, resulting in an average improvement of 2.3% (up to 24.1%) over vanilla PoT across 189 dataset combinations.
Prompt Transfer (PoT) is a recently-proposed approach to improve prompt-tuning, by initializing the target prompt with the existing prompt trained on similar source tasks. However, such a vanilla PoT approach usually achieves sub-optimal performance, as (i) the PoT is sensitive to the similarity of source-target pair and (ii) directly fine-tuning the prompt initialized with source prompt on target task might lead to forgetting of the useful general knowledge learned from source task. To tackle these issues, we propose a new metric to accurately predict the prompt transferability (regarding (i)), and a novel PoT approach (namely PANDA) that leverages the knowledge distillation technique to alleviate the knowledge forgetting effectively (regarding (ii)). Extensive and systematic experiments on 189 combinations of 21 source and 9 target datasets across 5 scales of PLMs demonstrate that: 1) our proposed metric works well to predict the prompt transferability; 2) our PANDA consistently outperforms the vanilla PoT approach by 2.3% average score (up to 24.1%) among all tasks and model sizes; 3) with our PANDA approach, prompt-tuning can achieve competitive and even better performance than model-tuning in various PLM scales scenarios. We have publicly released our code in https://github.com/WHU-ZQH/PANDA.