Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models
This work addresses the problem of prompt transferability for NLP practitioners, offering an incremental improvement in adapting large language models to tasks.
The paper tackles the challenge of transferring continuous prompts between different language models by proposing a zero-shot method that encodes source prompts into relative space and searches for corresponding target prompts, achieving effective generalization of task semantics across models and further enhancing transferability by combining semantics from multiple sources.
Prompt tuning in natural language processing (NLP) has become an increasingly popular method for adapting large language models to specific tasks. However, the transferability of these prompts, especially continuous prompts, between different models remains a challenge. In this work, we propose a zero-shot continuous prompt transfer method, where source prompts are encoded into relative space and the corresponding target prompts are searched for transferring to target models. Experimental results confirm the effectiveness of our method, showing that 'task semantics' in continuous prompts can be generalized across various language models. Moreover, we find that combining 'task semantics' from multiple source models can further enhance the generalizability of transfer.