Improving Input-label Mapping with Demonstration Replay for In-context Learning
This work addresses a specific bottleneck in ICL for NLP tasks, offering an incremental improvement by modifying attention mechanisms without training.
The paper tackles the limitation of causal attention in in-context learning (ICL) for large language models, which restricts input-label mapping by only allowing backward attention, and proposes a method called Repeated Demonstration with Sliding Causal Attention (RdSca) that duplicates and repositions demonstrations to improve this mapping, achieving significant performance gains as shown in experimental results.
In-context learning (ICL) is an emerging capability of large autoregressive language models where a few input-label demonstrations are appended to the input to enhance the model's understanding of downstream NLP tasks, without directly adjusting the model parameters. The effectiveness of ICL can be attributed to the strong language modeling capabilities of large language models (LLMs), which enable them to learn the mapping between input and labels based on in-context demonstrations. Despite achieving promising results, the causal nature of language modeling in ICL restricts the attention to be backward only, i.e., a token only attends to its previous tokens, failing to capture the full input-label information and limiting the model's performance. In this paper, we propose a novel ICL method called Repeated Demonstration with Sliding Causal Attention, (RdSca). Specifically, we duplicate later demonstrations and concatenate them to the front, allowing the model to `observe' the later information even under the causal restriction. Besides, we introduce sliding causal attention, which customizes causal attention to avoid information leakage. Experimental results show that our method significantly improves the input-label mapping in ICL demonstrations. We also conduct an in-depth analysis of how to customize the causal attention without training, which has been an unexplored area in previous research.