CrossICL: Cross-Task In-Context Learning via Unsupervised Demonstration Transfer
This addresses the challenge of obtaining demonstrations for users unwilling or unable to provide them, offering an incremental improvement in automating in-context learning.
The paper tackles the problem of reducing manual effort for in-context learning demonstrations by proposing CrossICL, a method to transfer demonstrations from source to target tasks, and demonstrates its effectiveness across 875 NLP tasks and six LLMs including GPT-4o.
In-Context Learning (ICL) enhances the performance of large language models (LLMs) with demonstrations. However, obtaining these demonstrations primarily relies on manual effort. In most real-world scenarios, users are often unwilling or unable to provide such demonstrations. Inspired by the human analogy, we explore a new ICL paradigm CrossICL to study how to utilize existing source task demonstrations in the ICL for target tasks, thereby obtaining reliable guidance without any additional manual effort. To explore this, we first design a two-stage alignment strategy to mitigate the interference caused by gaps across tasks, as the foundation for our experimental exploration. Based on it, we conduct comprehensive exploration of CrossICL, with 875 NLP tasks from the Super-NI benchmark and six types of LLMs, including GPT-4o. Experimental results demonstrate the effectiveness of CrossICL and provide valuable insights on questions like the criteria for selecting cross-task demonstrations, as well as the types of task-gap-induced interference in CrossICL.