Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator
This addresses the problem of data dependency in in-context learning for NLP practitioners, offering a more consistent and efficient approach, though it is incremental as it builds on existing in-context learning methods.
The paper tackles the reliance of in-context learning on external datasets by proposing self-generated in-context learning (SG-ICL), which generates demonstrations from the language model itself, resulting in performance significantly better than zero-shot learning and equivalent to about 0.6 gold training samples across four text classification tasks.
Large-scale pre-trained language models (PLMs) are well-known for being capable of solving a task simply by conditioning a few input-label pairs dubbed demonstrations on a prompt without being explicitly tuned for the desired downstream task. Such a process (i.e., in-context learning), however, naturally leads to high reliance on the demonstrations which are usually selected from external datasets. In this paper, we propose self-generated in-context learning (SG-ICL), which generates demonstrations for in-context learning from PLM itself to minimize the reliance on the external demonstration. We conduct experiments on four different text classification tasks and show SG-ICL significantly outperforms zero-shot learning and is generally worth approximately 0.6 gold training samples. Moreover, our generated demonstrations show more consistent performance with low variance compared to randomly selected demonstrations from the training dataset.