Technical Report -- Competition Solution for Prompt Tuning using Pretrained Language Model
This work addresses the challenge of fine-tuning models without access to gradients for users of Language Model as a Service, but it is incremental as it builds on existing methods.
The authors tackled the problem of adapting large pretrained language models to downstream tasks via prompt tuning in a black-box setting, achieving competitive results through modifications to BBTv2 and ensemble methods.
Prompt tuning recently becomes a hot-spot in the applications of large pretrained language models on specific downstream tasks. Regarding the Language Model as a Service (LMaaS), black-box tuning using derivative-free optimization (DFO) provides a novel approach to expand the practical scenarios of pretrained models and enrich the researches of few-shot learning. In this report, we present our solution in this competition that is based on the LMaaS scenario. Our solution consists of several modifications to BBTv2, including multiple label words, selection of P0, rolling update strategy, multi-task loss from MLP classifier, and finally using the ensemble method to further improve generalization ability. We also shared some strategies that we tried but didn't use in the final submission for further discussion. In the end we raised a question about the SNLI dataset and the impact on the results, as well as our concerns about the competition.