Semantic-Oriented Unlabeled Priming for Large-Scale Language Models
This addresses the challenge of high finetuning costs for large language models by enabling the use of abundant unlabeled data, though it is incremental as it builds on existing in-context learning methods.
The paper tackles the problem of adapting large language models to tasks without finetuning by leveraging unlabeled data, introducing Semantic-Oriented Unlabeled Priming (SOUP) and bag-of-contexts priming, which improve zero-shot performance by up to 5.2% on benchmark datasets.
Due to the high costs associated with finetuning large language models, various recent works propose to adapt them to specific tasks without any parameter updates through in-context learning. Unfortunately, for in-context learning there is currently no way to leverage unlabeled data, which is often much easier to obtain in large quantities than labeled examples. In this work, we therefore investigate ways to make use of unlabeled examples to improve the zero-shot performance of pretrained language models without any finetuning: We introduce Semantic-Oriented Unlabeled Priming (SOUP), a method that classifies examples by retrieving semantically similar unlabeled examples, assigning labels to them in a zero-shot fashion, and then using them for in-context learning. We also propose bag-of-contexts priming, a new priming strategy that is more suitable for our setting and enables the usage of more examples than fit into the context window.