Exploring Lottery Prompts for Pre-trained Language Models
This work addresses the computational burden of adapting large language models for NLP tasks, offering a more efficient alternative to fine-tuning, though it is incremental as it builds on existing prompting methods.
The paper tackles the inefficiency of fine-tuning pre-trained language models by exploring instance-specific 'lottery prompts' that can induce correct predictions, finding that these prompts can be generalized to unseen data with prompt ensembling, achieving results comparable to other gradient-free and optimization-free baselines.
Consistently scaling pre-trained language models (PLMs) imposes substantial burdens on model adaptation, necessitating more efficient alternatives to conventional fine-tuning. Given the advantage of prompting in the zero-shot setting and the observed performance fluctuation among different prompts, we explore the instance-level prompt and their generalizability. By searching through the prompt space, we first validate the assumption that for every instance, there is almost always a lottery prompt that induces the correct prediction from the PLM, and such prompt can be obtained at a low cost thanks to the inherent ability of PLMs. Meanwhile, we find that some strong lottery prompts have high performance over the whole training set, and they are equipped with distinguishable linguistic features. Lastly, we attempt to generalize the searched strong lottery prompts to unseen data with prompt ensembling method without any parameter tuning. Experiments are conducted on various types of NLP classification tasks and demonstrate that the proposed method can achieve comparable results with other gradient-free and optimization-free baselines.